2024-09-14T05:22:45.573 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-14T05:22:45.639 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-14T05:22:45.730 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-10_20:16:17-rbd-main-distro-default-smithi/7900324 branch: main description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zstd policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} email: ceph-qa@ceph.com first_in_suite: false job_id: '7900324' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: run two ceph clusters and install rbd-mirror - desc: configure the permissions for client.mirror - desc: 2 ceph clusters with 1 mon and 3 osds each - desc: run four rbd-mirror daemons per cluster - desc: run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon name: teuthology-2024-09-10_20:16:17-rbd-main-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 30 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug rbd: 20 debug rbd_mirror: 15 log to stderr: false rbd default features: 125 rbd mirror image policy type: none client.mirror.0: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.1: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.2: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.3: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluestore block size: 96636764160 bluestore compression algorithm: zstd bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: 7f24e589328ae9e639c0009d93340e5b235d0a47 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 7f24e589328ae9e639c0009d93340e5b235d0a47 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 0d61375b8ab53be4c233f5b93e463edfdff9316a owner: scheduled_teuthology@teuthology priority: 950 repo: https://git.ceph.com/ceph.git roles: - - cluster1.mon.a - cluster1.mgr.x - cluster2.mgr.x - cluster1.osd.0 - cluster1.osd.1 - cluster1.osd.2 - cluster1.client.0 - cluster2.client.0 - - cluster2.mon.a - cluster2.osd.0 - cluster2.osd.1 - cluster2.osd.2 - cluster1.client.mirror - cluster1.client.mirror.0 - cluster1.client.mirror.1 - cluster1.client.mirror.2 - cluster1.client.mirror.3 - cluster1.client.mirror.4 - cluster1.client.mirror.5 - cluster1.client.mirror.6 - cluster2.client.mirror - cluster2.client.mirror.0 - cluster2.client.mirror.1 - cluster2.client.mirror.2 - cluster2.client.mirror.3 - cluster2.client.mirror.4 - cluster2.client.mirror.5 - cluster2.client.mirror.6 seed: 5839 sha1: 7f24e589328ae9e639c0009d93340e5b235d0a47 sleep_before_teardown: 0 subset: 99/128 suite: rbd suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_0d61375b8ab53be4c233f5b93e463edfdff9316a/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 0d61375b8ab53be4c233f5b93e463edfdff9316a targets: smithi143.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBKp8YEKbvZo4dBiDX4/t6S8Va2K3MQZi0ygsc4Dx7PCp5DriXj2KiaYHnovMOotaCtpOJyPkhsJB6V3toaXXbo= smithi191.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL2v073v2G1I34ew/oySIIhRcoJk/PRN/EB4kIRX8Gp3X0OidBnpvoZx252VCP35d73k0EnPO2qJdEPgW68DolU= tasks: - install: extra_packages: - rbd-mirror - ceph: cluster: cluster1 - ceph: cluster: cluster2 - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - rbd-mirror: client: cluster1.client.mirror.0 thrash: true - rbd-mirror: client: cluster1.client.mirror.1 thrash: true - rbd-mirror: client: cluster1.client.mirror.2 thrash: true - rbd-mirror: client: cluster1.client.mirror.3 thrash: true - rbd-mirror: client: cluster2.client.mirror.0 thrash: true - rbd-mirror: client: cluster2.client.mirror.1 thrash: true - rbd-mirror: client: cluster2.client.mirror.2 thrash: true - rbd-mirror: client: cluster2.client.mirror.3 thrash: true - rbd-mirror-thrash: cluster: cluster1 - rbd-mirror-thrash: cluster: cluster2 - workunit: clients: cluster1.client.mirror: - rbd/rbd_mirror_stress.sh env: CEPH_ARGS: '' MIRROR_POOL_MODE: image RBD_IMAGE_FEATURES: layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_INSTANCES: '4' RBD_MIRROR_MODE: snapshot RBD_MIRROR_USE_EXISTING_CLUSTER: '1' RBD_MIRROR_USE_RBD_MIRROR: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-10_20:16:17 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-14T05:22:45.730 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_0d61375b8ab53be4c233f5b93e463edfdff9316a/qa; will attempt to use it 2024-09-14T05:22:45.731 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_0d61375b8ab53be4c233f5b93e463edfdff9316a/qa/tasks 2024-09-14T05:22:45.731 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-14T05:22:45.733 INFO:teuthology.task.internal:Checking packages... 2024-09-14T05:22:45.757 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '7f24e589328ae9e639c0009d93340e5b235d0a47' 2024-09-14T05:22:45.757 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-14T05:22:45.757 INFO:teuthology.packaging:ref: None 2024-09-14T05:22:45.757 INFO:teuthology.packaging:tag: None 2024-09-14T05:22:45.757 INFO:teuthology.packaging:branch: main 2024-09-14T05:22:45.757 INFO:teuthology.packaging:sha1: 7f24e589328ae9e639c0009d93340e5b235d0a47 2024-09-14T05:22:45.758 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-09-14T05:22:45.978 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4926.g1c941a66 2024-09-14T05:22:45.980 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-14T05:22:45.986 INFO:teuthology.task.internal:no buildpackages task found 2024-09-14T05:22:45.986 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-14T05:22:45.992 INFO:teuthology.task.internal:Saving configuration 2024-09-14T05:22:46.010 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-14T05:22:46.016 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-14T05:22:46.041 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi143.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-10_20:16:17-rbd-main-distro-default-smithi/7900324', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-14 05:17:57.011076', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBKp8YEKbvZo4dBiDX4/t6S8Va2K3MQZi0ygsc4Dx7PCp5DriXj2KiaYHnovMOotaCtpOJyPkhsJB6V3toaXXbo='} 2024-09-14T05:22:46.066 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi191.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-10_20:16:17-rbd-main-distro-default-smithi/7900324', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-14 05:17:57.012739', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL2v073v2G1I34ew/oySIIhRcoJk/PRN/EB4kIRX8Gp3X0OidBnpvoZx252VCP35d73k0EnPO2qJdEPgW68DolU='} 2024-09-14T05:22:46.066 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-14T05:22:46.072 INFO:teuthology.task.internal:roles: ubuntu@smithi143.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2024-09-14T05:22:46.073 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2024-09-14T05:22:46.073 INFO:teuthology.run_tasks:Running task console_log... 2024-09-14T05:22:46.130 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4220863be0>, signals=[15]) 2024-09-14T05:22:46.131 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-14T05:22:46.136 INFO:teuthology.task.internal:Opening connections... 2024-09-14T05:22:46.137 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi143.front.sepia.ceph.com 2024-09-14T05:22:46.138 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-14T05:22:46.216 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi191.front.sepia.ceph.com 2024-09-14T05:22:46.217 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-14T05:22:46.299 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-14T05:22:46.305 DEBUG:teuthology.orchestra.run.smithi143:> uname -m 2024-09-14T05:22:46.324 INFO:teuthology.orchestra.run.smithi143.stdout:x86_64 2024-09-14T05:22:46.325 DEBUG:teuthology.orchestra.run.smithi143:> cat /etc/os-release 2024-09-14T05:22:46.383 INFO:teuthology.orchestra.run.smithi143.stdout:NAME="CentOS Stream" 2024-09-14T05:22:46.383 INFO:teuthology.orchestra.run.smithi143.stdout:VERSION="9" 2024-09-14T05:22:46.383 INFO:teuthology.orchestra.run.smithi143.stdout:ID="centos" 2024-09-14T05:22:46.383 INFO:teuthology.orchestra.run.smithi143.stdout:ID_LIKE="rhel fedora" 2024-09-14T05:22:46.383 INFO:teuthology.orchestra.run.smithi143.stdout:VERSION_ID="9" 2024-09-14T05:22:46.384 INFO:teuthology.orchestra.run.smithi143.stdout:PLATFORM_ID="platform:el9" 2024-09-14T05:22:46.384 INFO:teuthology.orchestra.run.smithi143.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-14T05:22:46.384 INFO:teuthology.orchestra.run.smithi143.stdout:ANSI_COLOR="0;31" 2024-09-14T05:22:46.384 INFO:teuthology.orchestra.run.smithi143.stdout:LOGO="fedora-logo-icon" 2024-09-14T05:22:46.384 INFO:teuthology.orchestra.run.smithi143.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-14T05:22:46.384 INFO:teuthology.orchestra.run.smithi143.stdout:HOME_URL="https://centos.org/" 2024-09-14T05:22:46.384 INFO:teuthology.orchestra.run.smithi143.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-14T05:22:46.384 INFO:teuthology.orchestra.run.smithi143.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-14T05:22:46.384 INFO:teuthology.orchestra.run.smithi143.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-14T05:22:46.385 INFO:teuthology.lock.ops:Updating smithi143.front.sepia.ceph.com on lock server 2024-09-14T05:22:46.411 DEBUG:teuthology.orchestra.run.smithi191:> uname -m 2024-09-14T05:22:46.430 INFO:teuthology.orchestra.run.smithi191.stdout:x86_64 2024-09-14T05:22:46.430 DEBUG:teuthology.orchestra.run.smithi191:> cat /etc/os-release 2024-09-14T05:22:46.487 INFO:teuthology.orchestra.run.smithi191.stdout:NAME="CentOS Stream" 2024-09-14T05:22:46.487 INFO:teuthology.orchestra.run.smithi191.stdout:VERSION="9" 2024-09-14T05:22:46.487 INFO:teuthology.orchestra.run.smithi191.stdout:ID="centos" 2024-09-14T05:22:46.488 INFO:teuthology.orchestra.run.smithi191.stdout:ID_LIKE="rhel fedora" 2024-09-14T05:22:46.488 INFO:teuthology.orchestra.run.smithi191.stdout:VERSION_ID="9" 2024-09-14T05:22:46.488 INFO:teuthology.orchestra.run.smithi191.stdout:PLATFORM_ID="platform:el9" 2024-09-14T05:22:46.488 INFO:teuthology.orchestra.run.smithi191.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-14T05:22:46.488 INFO:teuthology.orchestra.run.smithi191.stdout:ANSI_COLOR="0;31" 2024-09-14T05:22:46.488 INFO:teuthology.orchestra.run.smithi191.stdout:LOGO="fedora-logo-icon" 2024-09-14T05:22:46.488 INFO:teuthology.orchestra.run.smithi191.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-14T05:22:46.488 INFO:teuthology.orchestra.run.smithi191.stdout:HOME_URL="https://centos.org/" 2024-09-14T05:22:46.488 INFO:teuthology.orchestra.run.smithi191.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-14T05:22:46.488 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-14T05:22:46.488 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-14T05:22:46.489 INFO:teuthology.lock.ops:Updating smithi191.front.sepia.ceph.com on lock server 2024-09-14T05:22:46.511 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-14T05:22:46.527 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-14T05:22:46.533 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-14T05:22:46.534 DEBUG:teuthology.orchestra.run.smithi143:> test '!' -e /home/ubuntu/cephtest 2024-09-14T05:22:46.536 DEBUG:teuthology.orchestra.run.smithi191:> test '!' -e /home/ubuntu/cephtest 2024-09-14T05:22:46.553 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-14T05:22:46.559 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-14T05:22:46.559 DEBUG:teuthology.orchestra.run.smithi143:> test -z $(ls -A /var/lib/ceph) 2024-09-14T05:22:46.596 DEBUG:teuthology.orchestra.run.smithi191:> test -z $(ls -A /var/lib/ceph) 2024-09-14T05:22:46.697 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-14T05:22:46.751 INFO:teuthology.run_tasks:Running task kernel... 2024-09-14T05:22:46.766 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-14T05:22:46.766 INFO:teuthology.task.kernel:config {'cluster1.mon.a': {'kdb': 1, 'sha1': 'distro'}, 'cluster2.mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-14T05:22:46.766 DEBUG:teuthology.orchestra.run.smithi143:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-14T05:22:46.767 DEBUG:teuthology.orchestra.run.smithi191:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-14T05:22:46.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T05:22:46.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T05:22:46.785 DEBUG:teuthology.orchestra.run.smithi143:> uname -r 2024-09-14T05:22:46.785 DEBUG:teuthology.orchestra.run.smithi191:> uname -r 2024-09-14T05:22:46.842 INFO:teuthology.orchestra.run.smithi191.stdout:5.14.0-503.el9.x86_64 2024-09-14T05:22:46.843 INFO:teuthology.task.kernel:Running kernel on smithi191: 5.14.0-503.el9.x86_64 2024-09-14T05:22:46.843 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum install -y kernel 2024-09-14T05:22:46.844 INFO:teuthology.orchestra.run.smithi143.stdout:5.14.0-503.el9.x86_64 2024-09-14T05:22:46.844 INFO:teuthology.task.kernel:Running kernel on smithi143: 5.14.0-503.el9.x86_64 2024-09-14T05:22:46.844 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum install -y kernel 2024-09-14T05:22:49.447 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.2 MB 00:00 2024-09-14T05:22:49.494 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS Stream 9 - BaseOS 9.5 MB/s | 8.2 MB 00:00 2024-09-14T05:22:51.503 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS Stream 9 - AppStream 23 MB/s | 20 MB 00:00 2024-09-14T05:22:54.848 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - AppStream 4.7 MB/s | 20 MB 00:04 2024-09-14T05:22:56.080 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS Stream 9 - CRB 8.1 MB/s | 6.5 MB 00:00 2024-09-14T05:22:57.580 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS Stream 9 - Extras packages 64 kB/s | 19 kB 00:00 2024-09-14T05:22:58.551 INFO:teuthology.orchestra.run.smithi143.stdout:Extra Packages for Enterprise Linux 27 MB/s | 23 MB 00:00 2024-09-14T05:22:59.091 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.5 MB 00:00 2024-09-14T05:23:00.617 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - Extras packages 61 kB/s | 19 kB 00:00 2024-09-14T05:23:01.705 INFO:teuthology.orchestra.run.smithi191.stdout:Extra Packages for Enterprise Linux 23 MB/s | 23 MB 00:00 2024-09-14T05:23:04.007 INFO:teuthology.orchestra.run.smithi143.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-09-14T05:23:06.055 INFO:teuthology.orchestra.run.smithi143.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-14T05:23:06.055 INFO:teuthology.orchestra.run.smithi143.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-14T05:23:06.139 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-09-14T05:23:06.150 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-09-14T05:23:06.150 INFO:teuthology.orchestra.run.smithi143.stdout: Package Architecture Version Repository Size 2024-09-14T05:23:06.151 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-09-14T05:23:06.151 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-09-14T05:23:06.151 INFO:teuthology.orchestra.run.smithi143.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-14T05:23:06.151 INFO:teuthology.orchestra.run.smithi143.stdout:Installing dependencies: 2024-09-14T05:23:06.151 INFO:teuthology.orchestra.run.smithi143.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-14T05:23:06.151 INFO:teuthology.orchestra.run.smithi143.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-14T05:23:06.151 INFO:teuthology.orchestra.run.smithi143.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-14T05:23:06.151 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:23:06.151 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-09-14T05:23:06.151 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-09-14T05:23:06.151 INFO:teuthology.orchestra.run.smithi143.stdout:Install 4 Packages 2024-09-14T05:23:06.151 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:23:06.152 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 79 M 2024-09-14T05:23:06.152 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 126 M 2024-09-14T05:23:06.152 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-09-14T05:23:06.341 INFO:teuthology.orchestra.run.smithi143.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 301 kB/s | 23 kB 00:00 2024-09-14T05:23:07.050 INFO:teuthology.orchestra.run.smithi143.stdout:(2/4): kernel-modules-core-5.14.0-505.el9.x86_6 40 MB/s | 29 MB 00:00 2024-09-14T05:23:07.144 INFO:teuthology.orchestra.run.smithi191.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-14T05:23:07.733 INFO:teuthology.orchestra.run.smithi143.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 24 MB/s | 35 MB 00:01 2024-09-14T05:23:08.284 INFO:teuthology.orchestra.run.smithi143.stdout:(4/4): kernel-core-5.14.0-505.el9.x86_64.rpm 7.8 MB/s | 16 MB 00:02 2024-09-14T05:23:08.284 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-09-14T05:23:08.284 INFO:teuthology.orchestra.run.smithi143.stdout:Total 37 MB/s | 79 MB 00:02 2024-09-14T05:23:08.967 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-09-14T05:23:09.210 INFO:teuthology.orchestra.run.smithi191.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-14T05:23:09.210 INFO:teuthology.orchestra.run.smithi191.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-14T05:23:09.295 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-14T05:23:09.301 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-09-14T05:23:09.301 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-09-14T05:23:09.307 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-14T05:23:09.307 INFO:teuthology.orchestra.run.smithi191.stdout: Package Architecture Version Repository Size 2024-09-14T05:23:09.308 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-14T05:23:09.308 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-09-14T05:23:09.308 INFO:teuthology.orchestra.run.smithi191.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-14T05:23:09.308 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-09-14T05:23:09.308 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-14T05:23:09.308 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-14T05:23:09.308 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-14T05:23:09.308 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:23:09.308 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-09-14T05:23:09.308 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-14T05:23:09.308 INFO:teuthology.orchestra.run.smithi191.stdout:Install 4 Packages 2024-09-14T05:23:09.308 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:23:09.309 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 79 M 2024-09-14T05:23:09.309 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 126 M 2024-09-14T05:23:09.309 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-09-14T05:23:09.496 INFO:teuthology.orchestra.run.smithi191.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 329 kB/s | 23 kB 00:00 2024-09-14T05:23:10.046 INFO:teuthology.orchestra.run.smithi191.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 25 MB/s | 16 MB 00:00 2024-09-14T05:23:10.272 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-09-14T05:23:10.272 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-09-14T05:23:10.714 INFO:teuthology.orchestra.run.smithi191.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 27 MB/s | 35 MB 00:01 2024-09-14T05:23:10.997 INFO:teuthology.orchestra.run.smithi191.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 19 MB/s | 29 MB 00:01 2024-09-14T05:23:10.997 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-09-14T05:23:10.998 INFO:teuthology.orchestra.run.smithi191.stdout:Total 47 MB/s | 79 MB 00:01 2024-09-14T05:23:11.686 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-09-14T05:23:12.043 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-09-14T05:23:12.043 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-09-14T05:23:12.727 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-09-14T05:23:13.029 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-09-14T05:23:13.030 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-09-14T05:23:13.479 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-14T05:23:13.563 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-14T05:23:15.481 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-09-14T05:23:16.152 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-14T05:23:16.238 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-14T05:23:16.366 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-14T05:23:16.496 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-14T05:23:19.028 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-14T05:23:19.176 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-14T05:23:21.333 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-14T05:23:21.371 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-14T05:23:23.997 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-14T05:23:24.037 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-14T05:23:26.206 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-14T05:23:28.874 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-14T05:24:04.422 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-14T05:24:04.461 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-14T05:24:04.786 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-14T05:24:04.786 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-14T05:24:04.786 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-14T05:24:04.786 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-14T05:24:05.627 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-14T05:24:05.627 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:24:05.627 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-09-14T05:24:05.627 INFO:teuthology.orchestra.run.smithi143.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-14T05:24:05.627 INFO:teuthology.orchestra.run.smithi143.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-14T05:24:05.627 INFO:teuthology.orchestra.run.smithi143.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-14T05:24:05.627 INFO:teuthology.orchestra.run.smithi143.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-14T05:24:05.628 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:24:05.628 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-09-14T05:24:05.969 DEBUG:teuthology.orchestra.run.smithi143:> rpm -q kernel | sort -rV | head -n 1 2024-09-14T05:24:06.059 INFO:teuthology.orchestra.run.smithi143.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-14T05:24:06.059 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-14T05:24:06.059 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-14T05:24:06.060 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-14T05:24:06.060 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-14T05:24:06.060 INFO:teuthology.task.kernel:Installing distro kernel on cluster1.mon.a... 2024-09-14T05:24:06.060 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi143.front.sepia.ceph.com, path=None, version=distro) 2024-09-14T05:24:06.060 DEBUG:teuthology.orchestra.run.smithi143:> rpm -q kernel | sort -rV | head -n 1 2024-09-14T05:24:06.119 INFO:teuthology.orchestra.run.smithi143.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-14T05:24:06.120 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-14T05:24:06.120 DEBUG:teuthology.orchestra.run.smithi143:> sudo rpm -qi grub2-tools 2024-09-14T05:24:06.152 INFO:teuthology.orchestra.run.smithi143.stdout:Name : grub2-tools 2024-09-14T05:24:06.152 INFO:teuthology.orchestra.run.smithi143.stdout:Epoch : 1 2024-09-14T05:24:06.153 INFO:teuthology.orchestra.run.smithi143.stdout:Version : 2.06 2024-09-14T05:24:06.153 INFO:teuthology.orchestra.run.smithi143.stdout:Release : 82.el9 2024-09-14T05:24:06.153 INFO:teuthology.orchestra.run.smithi143.stdout:Architecture: x86_64 2024-09-14T05:24:06.153 INFO:teuthology.orchestra.run.smithi143.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-14T05:24:06.153 INFO:teuthology.orchestra.run.smithi143.stdout:Group : Unspecified 2024-09-14T05:24:06.153 INFO:teuthology.orchestra.run.smithi143.stdout:Size : 8274670 2024-09-14T05:24:06.153 INFO:teuthology.orchestra.run.smithi143.stdout:License : GPLv3+ 2024-09-14T05:24:06.153 INFO:teuthology.orchestra.run.smithi143.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-14T05:24:06.153 INFO:teuthology.orchestra.run.smithi143.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-14T05:24:06.153 INFO:teuthology.orchestra.run.smithi143.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-14T05:24:06.153 INFO:teuthology.orchestra.run.smithi143.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-14T05:24:06.153 INFO:teuthology.orchestra.run.smithi143.stdout:Packager : builder@centos.org 2024-09-14T05:24:06.153 INFO:teuthology.orchestra.run.smithi143.stdout:Vendor : CentOS 2024-09-14T05:24:06.154 INFO:teuthology.orchestra.run.smithi143.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-14T05:24:06.154 INFO:teuthology.orchestra.run.smithi143.stdout:Summary : Support tools for GRUB. 2024-09-14T05:24:06.154 INFO:teuthology.orchestra.run.smithi143.stdout:Description : 2024-09-14T05:24:06.154 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:24:06.154 INFO:teuthology.orchestra.run.smithi143.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-14T05:24:06.154 INFO:teuthology.orchestra.run.smithi143.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-14T05:24:06.154 INFO:teuthology.orchestra.run.smithi143.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-14T05:24:06.154 INFO:teuthology.orchestra.run.smithi143.stdout:hardware devices. 2024-09-14T05:24:06.154 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:24:06.154 INFO:teuthology.orchestra.run.smithi143.stdout:This subpackage provides tools for support of all platforms. 2024-09-14T05:24:06.155 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-14T05:24:06.155 INFO:teuthology.task.kernel:Updating grub on smithi143 to boot 5.14.0-505.el9.x86_64 2024-09-14T05:24:06.155 DEBUG:teuthology.orchestra.run.smithi143:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-14T05:24:06.882 INFO:teuthology.orchestra.run.smithi143.stderr:Generating grub configuration file ... 2024-09-14T05:24:07.245 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-14T05:24:07.286 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-14T05:24:07.589 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-14T05:24:07.589 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-14T05:24:07.589 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-14T05:24:07.589 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-14T05:24:08.172 INFO:teuthology.orchestra.run.smithi143.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-14T05:24:08.200 INFO:teuthology.orchestra.run.smithi143.stderr:done 2024-09-14T05:24:08.205 DEBUG:teuthology.orchestra.run.smithi143:> mktemp 2024-09-14T05:24:08.221 INFO:teuthology.orchestra.run.smithi143.stdout:/tmp/tmp.chGlJ04xAO 2024-09-14T05:24:08.221 DEBUG:teuthology.orchestra.run.smithi143:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.chGlJ04xAO 2024-09-14T05:24:08.287 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 0666 /tmp/tmp.chGlJ04xAO 2024-09-14T05:24:08.427 DEBUG:teuthology.orchestra.remote:smithi143:/tmp/tmp.chGlJ04xAO is 6KB 2024-09-14T05:24:08.447 DEBUG:teuthology.orchestra.run.smithi143:> rm -fr /tmp/tmp.chGlJ04xAO 2024-09-14T05:24:08.461 DEBUG:teuthology.orchestra.run.smithi143:> sudo /bin/ls /boot/loader/entries || true 2024-09-14T05:24:08.502 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-14T05:24:08.503 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:24:08.503 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-09-14T05:24:08.503 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-14T05:24:08.503 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-14T05:24:08.503 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-14T05:24:08.503 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-14T05:24:08.503 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:24:08.503 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-14T05:24:08.528 INFO:teuthology.orchestra.run.smithi143.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-14T05:24:08.529 INFO:teuthology.orchestra.run.smithi143.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-14T05:24:08.529 INFO:teuthology.orchestra.run.smithi143.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-14T05:24:08.529 INFO:teuthology.orchestra.run.smithi143.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-14T05:24:08.530 DEBUG:teuthology.orchestra.run.smithi143:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-14T05:24:08.686 DEBUG:teuthology.orchestra.run.smithi143:> sudo shutdown -r now 2024-09-14T05:24:08.834 DEBUG:teuthology.orchestra.run.smithi191:> rpm -q kernel | sort -rV | head -n 1 2024-09-14T05:24:08.929 INFO:teuthology.orchestra.run.smithi191.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-14T05:24:08.929 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-14T05:24:08.929 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-14T05:24:08.929 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-14T05:24:08.929 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-14T05:24:08.929 INFO:teuthology.task.kernel:Installing distro kernel on cluster2.mon.a... 2024-09-14T05:24:08.930 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi191.front.sepia.ceph.com, path=None, version=distro) 2024-09-14T05:24:08.930 DEBUG:teuthology.orchestra.run.smithi191:> rpm -q kernel | sort -rV | head -n 1 2024-09-14T05:24:08.988 INFO:teuthology.orchestra.run.smithi191.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-14T05:24:08.989 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-14T05:24:08.989 DEBUG:teuthology.orchestra.run.smithi191:> sudo rpm -qi grub2-tools 2024-09-14T05:24:09.021 INFO:teuthology.orchestra.run.smithi191.stdout:Name : grub2-tools 2024-09-14T05:24:09.021 INFO:teuthology.orchestra.run.smithi191.stdout:Epoch : 1 2024-09-14T05:24:09.021 INFO:teuthology.orchestra.run.smithi191.stdout:Version : 2.06 2024-09-14T05:24:09.021 INFO:teuthology.orchestra.run.smithi191.stdout:Release : 82.el9 2024-09-14T05:24:09.021 INFO:teuthology.orchestra.run.smithi191.stdout:Architecture: x86_64 2024-09-14T05:24:09.021 INFO:teuthology.orchestra.run.smithi191.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-14T05:24:09.021 INFO:teuthology.orchestra.run.smithi191.stdout:Group : Unspecified 2024-09-14T05:24:09.022 INFO:teuthology.orchestra.run.smithi191.stdout:Size : 8274670 2024-09-14T05:24:09.022 INFO:teuthology.orchestra.run.smithi191.stdout:License : GPLv3+ 2024-09-14T05:24:09.022 INFO:teuthology.orchestra.run.smithi191.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-14T05:24:09.022 INFO:teuthology.orchestra.run.smithi191.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-14T05:24:09.022 INFO:teuthology.orchestra.run.smithi191.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-14T05:24:09.022 INFO:teuthology.orchestra.run.smithi191.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-14T05:24:09.022 INFO:teuthology.orchestra.run.smithi191.stdout:Packager : builder@centos.org 2024-09-14T05:24:09.022 INFO:teuthology.orchestra.run.smithi191.stdout:Vendor : CentOS 2024-09-14T05:24:09.022 INFO:teuthology.orchestra.run.smithi191.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-14T05:24:09.022 INFO:teuthology.orchestra.run.smithi191.stdout:Summary : Support tools for GRUB. 2024-09-14T05:24:09.022 INFO:teuthology.orchestra.run.smithi191.stdout:Description : 2024-09-14T05:24:09.022 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:24:09.022 INFO:teuthology.orchestra.run.smithi191.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-14T05:24:09.022 INFO:teuthology.orchestra.run.smithi191.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-14T05:24:09.023 INFO:teuthology.orchestra.run.smithi191.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-14T05:24:09.023 INFO:teuthology.orchestra.run.smithi191.stdout:hardware devices. 2024-09-14T05:24:09.023 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:24:09.023 INFO:teuthology.orchestra.run.smithi191.stdout:This subpackage provides tools for support of all platforms. 2024-09-14T05:24:09.024 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-14T05:24:09.024 INFO:teuthology.task.kernel:Updating grub on smithi191 to boot 5.14.0-505.el9.x86_64 2024-09-14T05:24:09.024 DEBUG:teuthology.orchestra.run.smithi191:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-14T05:24:09.692 INFO:teuthology.orchestra.run.smithi191.stderr:Generating grub configuration file ... 2024-09-14T05:24:10.991 INFO:teuthology.orchestra.run.smithi191.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-14T05:24:11.018 INFO:teuthology.orchestra.run.smithi191.stderr:done 2024-09-14T05:24:11.021 DEBUG:teuthology.orchestra.run.smithi191:> mktemp 2024-09-14T05:24:11.036 INFO:teuthology.orchestra.run.smithi191.stdout:/tmp/tmp.lOYvQzuv23 2024-09-14T05:24:11.036 DEBUG:teuthology.orchestra.run.smithi191:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.lOYvQzuv23 2024-09-14T05:24:11.106 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 0666 /tmp/tmp.lOYvQzuv23 2024-09-14T05:24:11.248 DEBUG:teuthology.orchestra.remote:smithi191:/tmp/tmp.lOYvQzuv23 is 6KB 2024-09-14T05:24:11.296 DEBUG:teuthology.orchestra.run.smithi191:> rm -fr /tmp/tmp.lOYvQzuv23 2024-09-14T05:24:11.311 DEBUG:teuthology.orchestra.run.smithi191:> sudo /bin/ls /boot/loader/entries || true 2024-09-14T05:24:11.378 INFO:teuthology.orchestra.run.smithi191.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-14T05:24:11.378 INFO:teuthology.orchestra.run.smithi191.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-14T05:24:11.378 INFO:teuthology.orchestra.run.smithi191.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-14T05:24:11.378 INFO:teuthology.orchestra.run.smithi191.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-14T05:24:11.380 DEBUG:teuthology.orchestra.run.smithi191:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-14T05:24:11.527 DEBUG:teuthology.orchestra.run.smithi191:> sudo shutdown -r now 2024-09-14T05:24:38.713 INFO:teuthology.task.kernel:Checking client cluster1.mon.a for new kernel version... 2024-09-14T05:24:38.713 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi143.front.sepia.ceph.com' 2024-09-14T05:24:38.714 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-14T05:24:41.529 INFO:teuthology.task.kernel:Checking client cluster2.mon.a for new kernel version... 2024-09-14T05:24:41.530 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-09-14T05:24:41.530 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-14T05:25:38.770 DEBUG:teuthology.orchestra.remote:timed out 2024-09-14T05:25:41.533 DEBUG:teuthology.orchestra.remote:timed out 2024-09-14T05:25:47.777 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi143.front.sepia.ceph.com' 2024-09-14T05:25:47.778 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-14T05:25:48.193 DEBUG:teuthology.orchestra.run.smithi143:> true 2024-09-14T05:25:48.548 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi143.front.sepia.ceph.com' 2024-09-14T05:25:48.549 INFO:teuthology.task.kernel:Checking kernel version of cluster1.mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-14T05:25:48.549 DEBUG:teuthology.orchestra.run.smithi143:> uname -r 2024-09-14T05:25:48.563 INFO:teuthology.orchestra.run.smithi143.stdout:5.14.0-505.el9.x86_64 2024-09-14T05:25:48.564 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-14T05:25:48.564 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-14T05:25:48.564 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-14T05:25:49.564 INFO:teuthology.task.kernel:Enabling kdb on cluster1.mon.a... 2024-09-14T05:25:49.565 DEBUG:teuthology.orchestra.run.smithi143:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-14T05:25:49.696 INFO:teuthology.orchestra.run.smithi143.stdout:ttyS1 2024-09-14T05:25:49.725 DEBUG:teuthology.parallel:result is None 2024-09-14T05:25:50.534 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-09-14T05:25:50.535 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-14T05:25:50.955 DEBUG:teuthology.orchestra.run.smithi191:> true 2024-09-14T05:25:51.335 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-09-14T05:25:51.335 INFO:teuthology.task.kernel:Checking kernel version of cluster2.mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-14T05:25:51.336 DEBUG:teuthology.orchestra.run.smithi191:> uname -r 2024-09-14T05:25:51.351 INFO:teuthology.orchestra.run.smithi191.stdout:5.14.0-505.el9.x86_64 2024-09-14T05:25:51.351 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-14T05:25:51.351 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-14T05:25:51.351 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-14T05:25:52.352 INFO:teuthology.task.kernel:Enabling kdb on cluster2.mon.a... 2024-09-14T05:25:52.352 DEBUG:teuthology.orchestra.run.smithi191:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-14T05:25:52.431 INFO:teuthology.orchestra.run.smithi191.stdout:ttyS1 2024-09-14T05:25:52.450 DEBUG:teuthology.parallel:result is None 2024-09-14T05:25:52.450 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-14T05:25:52.458 INFO:teuthology.task.internal:Creating test directory... 2024-09-14T05:25:52.458 DEBUG:teuthology.orchestra.run.smithi143:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-14T05:25:52.461 DEBUG:teuthology.orchestra.run.smithi191:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-14T05:25:52.476 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-14T05:25:52.482 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-14T05:25:52.487 INFO:teuthology.task.internal:Creating archive directory... 2024-09-14T05:25:52.488 DEBUG:teuthology.orchestra.run.smithi143:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-14T05:25:52.517 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-14T05:25:52.559 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-14T05:25:52.564 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-14T05:25:52.565 DEBUG:teuthology.orchestra.run.smithi143:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-14T05:25:52.595 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-14T05:25:52.618 INFO:teuthology.orchestra.run.smithi143.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-14T05:25:52.625 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-14T05:25:52.627 INFO:teuthology.orchestra.run.smithi143.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-14T05:25:52.636 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-14T05:25:52.637 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-14T05:25:52.644 INFO:teuthology.task.internal:Configuring sudo... 2024-09-14T05:25:52.644 DEBUG:teuthology.orchestra.run.smithi143:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-14T05:25:52.672 DEBUG:teuthology.orchestra.run.smithi191:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-14T05:25:52.699 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-14T05:25:52.708 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-14T05:25:52.708 DEBUG:teuthology.orchestra.run.smithi143:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-14T05:25:52.737 DEBUG:teuthology.orchestra.run.smithi191:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-14T05:25:52.752 DEBUG:teuthology.orchestra.run.smithi143:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-14T05:25:52.834 DEBUG:teuthology.orchestra.run.smithi143:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-14T05:25:52.881 DEBUG:teuthology.orchestra.run.smithi143:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-14T05:25:52.961 DEBUG:teuthology.orchestra.run.smithi143:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-14T05:25:53.002 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-14T05:25:53.002 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-14T05:25:53.067 DEBUG:teuthology.orchestra.run.smithi191:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-14T05:25:53.105 DEBUG:teuthology.orchestra.run.smithi191:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-14T05:25:53.149 DEBUG:teuthology.orchestra.run.smithi191:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-14T05:25:53.226 DEBUG:teuthology.orchestra.run.smithi191:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-14T05:25:53.265 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-14T05:25:53.265 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-14T05:25:53.327 DEBUG:teuthology.orchestra.run.smithi143:> sudo service rsyslog restart 2024-09-14T05:25:53.329 DEBUG:teuthology.orchestra.run.smithi191:> sudo service rsyslog restart 2024-09-14T05:25:53.375 INFO:teuthology.orchestra.run.smithi143.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-14T05:25:53.421 INFO:teuthology.orchestra.run.smithi191.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-14T05:25:53.885 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-14T05:25:53.893 INFO:teuthology.task.internal:Starting timer... 2024-09-14T05:25:53.893 INFO:teuthology.run_tasks:Running task pcp... 2024-09-14T05:25:53.904 INFO:teuthology.run_tasks:Running task selinux... 2024-09-14T05:25:53.912 DEBUG:teuthology.orchestra.run.smithi143:> sudo service auditd rotate 2024-09-14T05:25:53.955 INFO:teuthology.orchestra.run.smithi143.stdout:Rotating logs: 2024-09-14T05:25:53.957 DEBUG:teuthology.orchestra.run.smithi191:> sudo service auditd rotate 2024-09-14T05:25:54.074 INFO:teuthology.orchestra.run.smithi191.stdout:Rotating logs: 2024-09-14T05:25:54.076 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-14T05:25:54.076 DEBUG:teuthology.orchestra.run.smithi143:> /usr/sbin/getenforce 2024-09-14T05:25:54.115 INFO:teuthology.orchestra.run.smithi143.stdout:Permissive 2024-09-14T05:25:54.115 DEBUG:teuthology.orchestra.run.smithi191:> /usr/sbin/getenforce 2024-09-14T05:25:54.136 INFO:teuthology.orchestra.run.smithi191.stdout:Permissive 2024-09-14T05:25:54.137 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi143.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi191.front.sepia.ceph.com': 'permissive'} 2024-09-14T05:25:54.137 DEBUG:teuthology.orchestra.run.smithi143:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-14T05:25:54.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T05:25:54.182 DEBUG:teuthology.orchestra.run.smithi191:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-14T05:25:54.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T05:25:54.207 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-14T05:25:54.207 DEBUG:teuthology.orchestra.run.smithi143:> sudo /usr/sbin/setenforce permissive 2024-09-14T05:25:54.254 DEBUG:teuthology.orchestra.run.smithi191:> sudo /usr/sbin/setenforce permissive 2024-09-14T05:25:54.282 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-14T05:25:54.291 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-14T05:25:54.294 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-14T05:25:54.411 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-14T05:25:54.424 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-14T05:25:54.425 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi143.front.sepia.ceph.com,smithi191.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-14T05:29:36.662 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi143.front.sepia.ceph.com'), Remote(name='ubuntu@smithi191.front.sepia.ceph.com')] 2024-09-14T05:29:36.664 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi143.front.sepia.ceph.com' 2024-09-14T05:29:36.665 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-14T05:29:36.742 DEBUG:teuthology.orchestra.run.smithi143:> true 2024-09-14T05:29:36.816 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi143.front.sepia.ceph.com' 2024-09-14T05:29:36.816 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-09-14T05:29:36.817 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-14T05:29:36.891 DEBUG:teuthology.orchestra.run.smithi191:> true 2024-09-14T05:29:36.926 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-09-14T05:29:36.926 INFO:teuthology.run_tasks:Running task clock... 2024-09-14T05:29:36.943 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-14T05:29:36.943 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-14T05:29:36.944 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-14T05:29:36.946 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-14T05:29:36.946 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-14T05:29:36.974 INFO:teuthology.orchestra.run.smithi143.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-14T05:29:36.988 INFO:teuthology.orchestra.run.smithi143.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-14T05:29:36.995 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-14T05:29:37.009 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-14T05:29:37.013 INFO:teuthology.orchestra.run.smithi143.stderr:sudo: ntpd: command not found 2024-09-14T05:29:37.024 INFO:teuthology.orchestra.run.smithi143.stdout:506 Cannot talk to daemon 2024-09-14T05:29:37.034 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: ntpd: command not found 2024-09-14T05:29:37.038 INFO:teuthology.orchestra.run.smithi143.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-14T05:29:37.045 INFO:teuthology.orchestra.run.smithi191.stdout:506 Cannot talk to daemon 2024-09-14T05:29:37.052 INFO:teuthology.orchestra.run.smithi143.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-14T05:29:37.060 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-14T05:29:37.074 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-14T05:29:37.107 INFO:teuthology.orchestra.run.smithi143.stderr:bash: line 1: ntpq: command not found 2024-09-14T05:29:37.112 INFO:teuthology.orchestra.run.smithi143.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-14T05:29:37.112 INFO:teuthology.orchestra.run.smithi143.stdout:=============================================================================== 2024-09-14T05:29:37.113 INFO:teuthology.orchestra.run.smithi143.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-14T05:29:37.113 INFO:teuthology.orchestra.run.smithi143.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-14T05:29:37.113 INFO:teuthology.orchestra.run.smithi143.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-14T05:29:37.113 INFO:teuthology.orchestra.run.smithi143.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-14T05:29:37.130 INFO:teuthology.orchestra.run.smithi191.stderr:bash: line 1: ntpq: command not found 2024-09-14T05:29:37.133 INFO:teuthology.orchestra.run.smithi191.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-14T05:29:37.133 INFO:teuthology.orchestra.run.smithi191.stdout:=============================================================================== 2024-09-14T05:29:37.133 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-14T05:29:37.133 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-14T05:29:37.134 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-14T05:29:37.134 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-14T05:29:37.135 INFO:teuthology.run_tasks:Running task install... 2024-09-14T05:29:37.142 DEBUG:teuthology.task.install:project ceph 2024-09-14T05:29:37.142 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '7f24e589328ae9e639c0009d93340e5b235d0a47'}} 2024-09-14T05:29:37.143 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '7f24e589328ae9e639c0009d93340e5b235d0a47'} 2024-09-14T05:29:37.143 INFO:teuthology.task.install:Using flavor: default 2024-09-14T05:29:37.148 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-14T05:29:37.148 INFO:teuthology.task.install:extra packages: ['rbd-mirror'] 2024-09-14T05:29:37.149 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '7f24e589328ae9e639c0009d93340e5b235d0a47', 'tag': None, 'wait_for_package': False} 2024-09-14T05:29:37.149 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7f24e589328ae9e639c0009d93340e5b235d0a47 2024-09-14T05:29:37.151 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '7f24e589328ae9e639c0009d93340e5b235d0a47', 'tag': None, 'wait_for_package': False} 2024-09-14T05:29:37.152 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7f24e589328ae9e639c0009d93340e5b235d0a47 2024-09-14T05:29:37.301 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/centos/9/flavors/default/ 2024-09-14T05:29:37.301 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4775.g7f24e589 2024-09-14T05:29:37.304 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/centos/9/flavors/default/ 2024-09-14T05:29:37.304 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4775.g7f24e589 2024-09-14T05:29:37.412 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-14T05:29:37.412 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-14T05:29:37.412 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-14T05:29:37.428 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/main/7f24e589328ae9e639c0009d93340e5b235d0a47/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-14T05:29:37.428 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-14T05:29:37.428 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-14T05:29:37.438 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2024-09-14T05:29:37.438 DEBUG:teuthology.orchestra.run.smithi143:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/7f24e589328ae9e639c0009d93340e5b235d0a47/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-14T05:29:37.454 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2024-09-14T05:29:37.454 DEBUG:teuthology.orchestra.run.smithi191:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/7f24e589328ae9e639c0009d93340e5b235d0a47/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-14T05:29:37.505 DEBUG:teuthology.orchestra.run.smithi143:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-14T05:29:37.521 DEBUG:teuthology.orchestra.run.smithi191:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-14T05:29:37.579 DEBUG:teuthology.orchestra.run.smithi143:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-14T05:29:37.596 DEBUG:teuthology.orchestra.run.smithi191:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-14T05:29:37.644 INFO:teuthology.orchestra.run.smithi143.stdout:check_obsoletes = 1 2024-09-14T05:29:37.645 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum clean all 2024-09-14T05:29:37.661 INFO:teuthology.orchestra.run.smithi191.stdout:check_obsoletes = 1 2024-09-14T05:29:37.663 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum clean all 2024-09-14T05:29:37.886 INFO:teuthology.orchestra.run.smithi143.stdout:45 files removed 2024-09-14T05:29:37.899 INFO:teuthology.orchestra.run.smithi191.stdout:45 files removed 2024-09-14T05:29:37.914 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph-radosgw 2024-09-14T05:29:37.926 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-radosgw 2024-09-14T05:29:38.493 INFO:teuthology.orchestra.run.smithi143.stdout:ceph packages for x86_64 285 kB/s | 84 kB 00:00 2024-09-14T05:29:38.802 INFO:teuthology.orchestra.run.smithi143.stdout:ceph noarch packages 91 kB/s | 24 kB 00:00 2024-09-14T05:29:39.086 INFO:teuthology.orchestra.run.smithi143.stdout:ceph source packages 7.8 kB/s | 1.9 kB 00:00 2024-09-14T05:29:40.686 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS Stream 9 - BaseOS 5.3 MB/s | 8.2 MB 00:01 2024-09-14T05:29:42.846 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS Stream 9 - AppStream 19 MB/s | 20 MB 00:01 2024-09-14T05:29:47.180 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.5 MB 00:00 2024-09-14T05:29:48.597 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS Stream 9 - Extras packages 103 kB/s | 19 kB 00:00 2024-09-14T05:29:49.702 INFO:teuthology.orchestra.run.smithi143.stdout:Extra Packages for Enterprise Linux 23 MB/s | 23 MB 00:00 2024-09-14T05:29:55.116 INFO:teuthology.orchestra.run.smithi143.stdout:lab-extras 39 kB/s | 1.7 kB 00:00 2024-09-14T05:29:57.510 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-09-14T05:29:57.525 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-09-14T05:29:57.525 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repository Size 2024-09-14T05:29:57.525 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-09-14T05:29:57.525 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-09-14T05:29:57.525 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-radosgw x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 11 M 2024-09-14T05:29:57.525 INFO:teuthology.orchestra.run.smithi143.stdout:Upgrading: 2024-09-14T05:29:57.525 INFO:teuthology.orchestra.run.smithi143.stdout: librados2 x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 3.5 M 2024-09-14T05:29:57.525 INFO:teuthology.orchestra.run.smithi143.stdout: librbd1 x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 3.2 M 2024-09-14T05:29:57.525 INFO:teuthology.orchestra.run.smithi143.stdout:Installing dependencies: 2024-09-14T05:29:57.525 INFO:teuthology.orchestra.run.smithi143.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-14T05:29:57.525 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-base x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 5.5 M 2024-09-14T05:29:57.525 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-common x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 22 M 2024-09-14T05:29:57.526 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-selinux x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 25 k 2024-09-14T05:29:57.526 INFO:teuthology.orchestra.run.smithi143.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-14T05:29:57.526 INFO:teuthology.orchestra.run.smithi143.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-14T05:29:57.526 INFO:teuthology.orchestra.run.smithi143.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-14T05:29:57.526 INFO:teuthology.orchestra.run.smithi143.stdout: libcephfs2 x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 755 k 2024-09-14T05:29:57.526 INFO:teuthology.orchestra.run.smithi143.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-14T05:29:57.526 INFO:teuthology.orchestra.run.smithi143.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-14T05:29:57.526 INFO:teuthology.orchestra.run.smithi143.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-14T05:29:57.526 INFO:teuthology.orchestra.run.smithi143.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-14T05:29:57.526 INFO:teuthology.orchestra.run.smithi143.stdout: libradosstriper1 x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 508 k 2024-09-14T05:29:57.526 INFO:teuthology.orchestra.run.smithi143.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-14T05:29:57.526 INFO:teuthology.orchestra.run.smithi143.stdout: librgw2 x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 5.5 M 2024-09-14T05:29:57.527 INFO:teuthology.orchestra.run.smithi143.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-14T05:29:57.527 INFO:teuthology.orchestra.run.smithi143.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-14T05:29:57.527 INFO:teuthology.orchestra.run.smithi143.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-14T05:29:57.527 INFO:teuthology.orchestra.run.smithi143.stdout: python3-ceph-argparse x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 45 k 2024-09-14T05:29:57.527 INFO:teuthology.orchestra.run.smithi143.stdout: python3-ceph-common x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 143 k 2024-09-14T05:29:57.527 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cephfs x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 161 k 2024-09-14T05:29:57.527 INFO:teuthology.orchestra.run.smithi143.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-14T05:29:57.527 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rados x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 321 k 2024-09-14T05:29:57.527 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rbd x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 306 k 2024-09-14T05:29:57.527 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rgw x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 99 k 2024-09-14T05:29:57.527 INFO:teuthology.orchestra.run.smithi143.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-14T05:29:57.527 INFO:teuthology.orchestra.run.smithi143.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-14T05:29:57.527 INFO:teuthology.orchestra.run.smithi143.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-14T05:29:57.527 INFO:teuthology.orchestra.run.smithi143.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-14T05:29:57.528 INFO:teuthology.orchestra.run.smithi143.stdout:Installing weak dependencies: 2024-09-14T05:29:57.528 INFO:teuthology.orchestra.run.smithi143.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-14T05:29:57.528 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:29:57.528 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-09-14T05:29:57.528 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-09-14T05:29:57.528 INFO:teuthology.orchestra.run.smithi143.stdout:Install 31 Packages 2024-09-14T05:29:57.528 INFO:teuthology.orchestra.run.smithi143.stdout:Upgrade 2 Packages 2024-09-14T05:29:57.528 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:29:57.528 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 62 M 2024-09-14T05:29:57.529 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-09-14T05:29:58.333 INFO:teuthology.orchestra.run.smithi143.stdout:(1/33): ceph-base-19.3.0-4775.g7f24e589.el9.x86 8.1 MB/s | 5.5 MB 00:00 2024-09-14T05:29:58.400 INFO:teuthology.orchestra.run.smithi143.stdout:(2/33): ceph-selinux-19.3.0-4775.g7f24e589.el9. 379 kB/s | 25 kB 00:00 2024-09-14T05:29:58.493 INFO:teuthology.orchestra.run.smithi143.stdout:(3/33): libcephfs2-19.3.0-4775.g7f24e589.el9.x8 8.0 MB/s | 755 kB 00:00 2024-09-14T05:29:58.609 INFO:teuthology.orchestra.run.smithi143.stdout:(4/33): libradosstriper1-19.3.0-4775.g7f24e589. 4.3 MB/s | 508 kB 00:00 2024-09-14T05:29:58.815 INFO:teuthology.orchestra.run.smithi191.stdout:ceph packages for x86_64 4.1 kB/s | 84 kB 00:20 2024-09-14T05:29:58.968 INFO:teuthology.orchestra.run.smithi143.stdout:(5/33): ceph-radosgw-19.3.0-4775.g7f24e589.el9. 8.3 MB/s | 11 MB 00:01 2024-09-14T05:29:59.026 INFO:teuthology.orchestra.run.smithi143.stdout:(6/33): python3-ceph-argparse-19.3.0-4775.g7f24 778 kB/s | 45 kB 00:00 2024-09-14T05:29:59.102 INFO:teuthology.orchestra.run.smithi143.stdout:(7/33): librgw2-19.3.0-4775.g7f24e589.el9.x86_6 11 MB/s | 5.5 MB 00:00 2024-09-14T05:29:59.127 INFO:teuthology.orchestra.run.smithi143.stdout:(8/33): python3-ceph-common-19.3.0-4775.g7f24e5 1.4 MB/s | 143 kB 00:00 2024-09-14T05:29:59.152 INFO:teuthology.orchestra.run.smithi143.stdout:(9/33): python3-cephfs-19.3.0-4775.g7f24e589.el 3.1 MB/s | 161 kB 00:00 2024-09-14T05:29:59.157 INFO:teuthology.orchestra.run.smithi191.stdout:ceph noarch packages 81 kB/s | 24 kB 00:00 2024-09-14T05:29:59.186 INFO:teuthology.orchestra.run.smithi143.stdout:(10/33): python3-rados-19.3.0-4775.g7f24e589.el 5.4 MB/s | 321 kB 00:00 2024-09-14T05:29:59.236 INFO:teuthology.orchestra.run.smithi143.stdout:(11/33): python3-rbd-19.3.0-4775.g7f24e589.el9. 3.6 MB/s | 306 kB 00:00 2024-09-14T05:29:59.253 INFO:teuthology.orchestra.run.smithi143.stdout:(12/33): python3-rgw-19.3.0-4775.g7f24e589.el9. 1.4 MB/s | 99 kB 00:00 2024-09-14T05:29:59.558 INFO:teuthology.orchestra.run.smithi191.stdout:ceph source packages 5.3 kB/s | 1.9 kB 00:00 2024-09-14T05:29:59.770 INFO:teuthology.orchestra.run.smithi143.stdout:(13/33): ceph-common-19.3.0-4775.g7f24e589.el9. 10 MB/s | 22 MB 00:02 2024-09-14T05:29:59.795 INFO:teuthology.orchestra.run.smithi143.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 191 kB/s | 107 kB 00:00 2024-09-14T05:29:59.820 INFO:teuthology.orchestra.run.smithi143.stdout:(15/33): libnbd-1.20.2-2.el9.x86_64.rpm 301 kB/s | 171 kB 00:00 2024-09-14T05:29:59.913 INFO:teuthology.orchestra.run.smithi143.stdout:(16/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 387 kB/s | 45 kB 00:00 2024-09-14T05:30:00.008 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - BaseOS 20 MB/s | 8.2 MB 00:00 2024-09-14T05:30:00.121 INFO:teuthology.orchestra.run.smithi143.stdout:(17/33): librdkafka-1.6.1-102.el9.x86_64.rpm 2.2 MB/s | 662 kB 00:00 2024-09-14T05:30:00.196 INFO:teuthology.orchestra.run.smithi143.stdout:(18/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.0 MB/s | 292 kB 00:00 2024-09-14T05:30:00.230 INFO:teuthology.orchestra.run.smithi143.stdout:(19/33): python3-prettytable-0.7.2-27.el9.noarc 389 kB/s | 42 kB 00:00 2024-09-14T05:30:00.389 INFO:teuthology.orchestra.run.smithi143.stdout:(20/33): qatlib-24.02.0-1.el9.x86_64.rpm 1.1 MB/s | 221 kB 00:00 2024-09-14T05:30:00.414 INFO:teuthology.orchestra.run.smithi143.stdout:(21/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 249 kB/s | 160 kB 00:00 2024-09-14T05:30:00.439 INFO:teuthology.orchestra.run.smithi143.stdout:(22/33): qatlib-service-24.02.0-1.el9.x86_64.rp 171 kB/s | 36 kB 00:00 2024-09-14T05:30:00.490 INFO:teuthology.orchestra.run.smithi143.stdout:(23/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 4.0 MB/s | 308 kB 00:00 2024-09-14T05:30:00.523 INFO:teuthology.orchestra.run.smithi143.stdout:(24/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 353 kB/s | 47 kB 00:00 2024-09-14T05:30:00.549 INFO:teuthology.orchestra.run.smithi143.stdout:(25/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 443 kB/s | 26 kB 00:00 2024-09-14T05:30:00.582 INFO:teuthology.orchestra.run.smithi143.stdout:(26/33): libunwind-1.6.2-1.el9.x86_64.rpm 2.5 MB/s | 67 kB 00:00 2024-09-14T05:30:00.616 INFO:teuthology.orchestra.run.smithi143.stdout:(27/33): liboath-2.6.7-2.el9.x86_64.rpm 527 kB/s | 49 kB 00:00 2024-09-14T05:30:00.649 INFO:teuthology.orchestra.run.smithi143.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-14T05:30:00.683 INFO:teuthology.orchestra.run.smithi143.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.3 MB/s | 839 kB 00:00 2024-09-14T05:30:00.750 INFO:teuthology.orchestra.run.smithi143.stdout:(30/33): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-09-14T05:30:00.817 INFO:teuthology.orchestra.run.smithi143.stdout:(31/33): libarrow-9.0.0-11.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-09-14T05:30:01.092 INFO:teuthology.orchestra.run.smithi143.stdout:(32/33): librados2-19.3.0-4775.g7f24e589.el9.x8 8.5 MB/s | 3.5 MB 00:00 2024-09-14T05:30:01.151 INFO:teuthology.orchestra.run.smithi143.stdout:(33/33): librbd1-19.3.0-4775.g7f24e589.el9.x86_ 8.1 MB/s | 3.2 MB 00:00 2024-09-14T05:30:01.156 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-09-14T05:30:01.156 INFO:teuthology.orchestra.run.smithi143.stdout:Total 17 MB/s | 62 MB 00:03 2024-09-14T05:30:01.277 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-09-14T05:30:01.313 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-09-14T05:30:01.314 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-09-14T05:30:01.996 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-09-14T05:30:01.997 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-09-14T05:30:02.109 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - AppStream 20 MB/s | 20 MB 00:01 2024-09-14T05:30:03.540 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-09-14T05:30:03.610 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-14T05:30:03.647 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/35 2024-09-14T05:30:03.672 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/35 2024-09-14T05:30:03.695 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-14T05:30:03.734 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-14T05:30:03.932 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-14T05:30:03.985 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-14T05:30:04.400 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/35 2024-09-14T05:30:04.466 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : librados2-2:19.3.0-4775.g7f24e589.el9.x86_64 7/35 2024-09-14T05:30:04.558 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: librados2-2:19.3.0-4775.g7f24e589.el9.x86_64 7/35 2024-09-14T05:30:04.648 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-rados-2:19.3.0-4775.g7f24e589.el9.x86_64 8/35 2024-09-14T05:30:04.676 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libcephfs2-2:19.3.0-4775.g7f24e589.el9.x86_64 9/35 2024-09-14T05:30:04.739 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: libcephfs2-2:19.3.0-4775.g7f24e589.el9.x86_64 9/35 2024-09-14T05:30:04.768 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/35 2024-09-14T05:30:04.811 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-ceph-argparse-2:19.3.0-4775.g7f24e589.el9. 11/35 2024-09-14T05:30:04.881 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-cephfs-2:19.3.0-4775.g7f24e589.el9.x86_64 12/35 2024-09-14T05:30:04.903 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libradosstriper1-2:19.3.0-4775.g7f24e589.el9.x86_6 13/35 2024-09-14T05:30:04.967 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4775.g7f24e589.el9.x86_6 13/35 2024-09-14T05:30:04.993 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/35 2024-09-14T05:30:05.102 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/35 2024-09-14T05:30:05.136 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/35 2024-09-14T05:30:05.184 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/35 2024-09-14T05:30:05.210 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/35 2024-09-14T05:30:05.239 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/35 2024-09-14T05:30:05.272 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-14T05:30:05.309 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/35 2024-09-14T05:30:05.627 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-14T05:30:05.677 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : librbd1-2:19.3.0-4775.g7f24e589.el9.x86_64 23/35 2024-09-14T05:30:05.773 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: librbd1-2:19.3.0-4775.g7f24e589.el9.x86_64 23/35 2024-09-14T05:30:05.814 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-rbd-2:19.3.0-4775.g7f24e589.el9.x86_64 24/35 2024-09-14T05:30:05.859 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-14T05:30:06.334 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - CRB 17 MB/s | 6.5 MB 00:00 2024-09-14T05:30:06.360 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-14T05:30:06.388 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : librgw2-2:19.3.0-4775.g7f24e589.el9.x86_64 27/35 2024-09-14T05:30:06.468 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: librgw2-2:19.3.0-4775.g7f24e589.el9.x86_64 27/35 2024-09-14T05:30:06.577 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-rgw-2:19.3.0-4775.g7f24e589.el9.x86_64 28/35 2024-09-14T05:30:06.615 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-ceph-common-2:19.3.0-4775.g7f24e589.el9.x8 29/35 2024-09-14T05:30:07.885 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - Extras packages 117 kB/s | 19 kB 00:00 2024-09-14T05:30:08.971 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-common-2:19.3.0-4775.g7f24e589.el9.x86_64 30/35 2024-09-14T05:30:09.038 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-common-2:19.3.0-4775.g7f24e589.el9.x86_64 30/35 2024-09-14T05:30:09.281 INFO:teuthology.orchestra.run.smithi191.stdout:Extra Packages for Enterprise Linux 18 MB/s | 23 MB 00:01 2024-09-14T05:30:09.607 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-common-2:19.3.0-4775.g7f24e589.el9.x86_64 30/35 2024-09-14T05:30:09.649 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-base-2:19.3.0-4775.g7f24e589.el9.x86_64 31/35 2024-09-14T05:30:09.740 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-base-2:19.3.0-4775.g7f24e589.el9.x86_64 31/35 2024-09-14T05:30:09.740 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-14T05:30:09.740 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-14T05:30:09.740 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:30:09.775 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-selinux-2:19.3.0-4775.g7f24e589.el9.x86_64 32/35 2024-09-14T05:30:14.754 INFO:teuthology.orchestra.run.smithi191.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-14T05:30:17.189 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-14T05:30:17.222 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-14T05:30:17.222 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-09-14T05:30:17.222 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-14T05:30:17.222 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-09-14T05:30:17.222 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-radosgw x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 11 M 2024-09-14T05:30:17.223 INFO:teuthology.orchestra.run.smithi191.stdout:Upgrading: 2024-09-14T05:30:17.223 INFO:teuthology.orchestra.run.smithi191.stdout: librados2 x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 3.5 M 2024-09-14T05:30:17.223 INFO:teuthology.orchestra.run.smithi191.stdout: librbd1 x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 3.2 M 2024-09-14T05:30:17.223 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-09-14T05:30:17.223 INFO:teuthology.orchestra.run.smithi191.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-14T05:30:17.223 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-base x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 5.5 M 2024-09-14T05:30:17.223 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-common x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 22 M 2024-09-14T05:30:17.223 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-selinux x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 25 k 2024-09-14T05:30:17.223 INFO:teuthology.orchestra.run.smithi191.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-14T05:30:17.223 INFO:teuthology.orchestra.run.smithi191.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-14T05:30:17.223 INFO:teuthology.orchestra.run.smithi191.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-14T05:30:17.223 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs2 x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 755 k 2024-09-14T05:30:17.224 INFO:teuthology.orchestra.run.smithi191.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-14T05:30:17.224 INFO:teuthology.orchestra.run.smithi191.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-14T05:30:17.224 INFO:teuthology.orchestra.run.smithi191.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-14T05:30:17.224 INFO:teuthology.orchestra.run.smithi191.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-14T05:30:17.224 INFO:teuthology.orchestra.run.smithi191.stdout: libradosstriper1 x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 508 k 2024-09-14T05:30:17.224 INFO:teuthology.orchestra.run.smithi191.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-14T05:30:17.224 INFO:teuthology.orchestra.run.smithi191.stdout: librgw2 x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 5.5 M 2024-09-14T05:30:17.224 INFO:teuthology.orchestra.run.smithi191.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-14T05:30:17.224 INFO:teuthology.orchestra.run.smithi191.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-14T05:30:17.224 INFO:teuthology.orchestra.run.smithi191.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-14T05:30:17.224 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-argparse x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 45 k 2024-09-14T05:30:17.224 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-common x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 143 k 2024-09-14T05:30:17.225 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cephfs x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 161 k 2024-09-14T05:30:17.225 INFO:teuthology.orchestra.run.smithi191.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-14T05:30:17.225 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rados x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 321 k 2024-09-14T05:30:17.225 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rbd x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 306 k 2024-09-14T05:30:17.225 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rgw x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 99 k 2024-09-14T05:30:17.225 INFO:teuthology.orchestra.run.smithi191.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-14T05:30:17.225 INFO:teuthology.orchestra.run.smithi191.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-14T05:30:17.225 INFO:teuthology.orchestra.run.smithi191.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-14T05:30:17.225 INFO:teuthology.orchestra.run.smithi191.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-14T05:30:17.225 INFO:teuthology.orchestra.run.smithi191.stdout:Installing weak dependencies: 2024-09-14T05:30:17.225 INFO:teuthology.orchestra.run.smithi191.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-14T05:30:17.225 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:30:17.225 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-09-14T05:30:17.226 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-14T05:30:17.226 INFO:teuthology.orchestra.run.smithi191.stdout:Install 31 Packages 2024-09-14T05:30:17.226 INFO:teuthology.orchestra.run.smithi191.stdout:Upgrade 2 Packages 2024-09-14T05:30:17.226 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:30:17.226 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 62 M 2024-09-14T05:30:17.226 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-09-14T05:30:18.238 INFO:teuthology.orchestra.run.smithi191.stdout:(1/33): ceph-base-19.3.0-4775.g7f24e589.el9.x86 9.9 MB/s | 5.5 MB 00:00 2024-09-14T05:30:18.297 INFO:teuthology.orchestra.run.smithi191.stdout:(2/33): ceph-selinux-19.3.0-4775.g7f24e589.el9. 432 kB/s | 25 kB 00:00 2024-09-14T05:30:18.464 INFO:teuthology.orchestra.run.smithi191.stdout:(3/33): ceph-radosgw-19.3.0-4775.g7f24e589.el9. 14 MB/s | 11 MB 00:00 2024-09-14T05:30:18.497 INFO:teuthology.orchestra.run.smithi191.stdout:(4/33): libcephfs2-19.3.0-4775.g7f24e589.el9.x8 3.7 MB/s | 755 kB 00:00 2024-09-14T05:30:18.539 INFO:teuthology.orchestra.run.smithi191.stdout:(5/33): libradosstriper1-19.3.0-4775.g7f24e589. 6.6 MB/s | 508 kB 00:00 2024-09-14T05:30:18.598 INFO:teuthology.orchestra.run.smithi191.stdout:(6/33): python3-ceph-argparse-19.3.0-4775.g7f24 772 kB/s | 45 kB 00:00 2024-09-14T05:30:18.681 INFO:teuthology.orchestra.run.smithi191.stdout:(7/33): python3-ceph-common-19.3.0-4775.g7f24e5 1.7 MB/s | 143 kB 00:00 2024-09-14T05:30:18.798 INFO:teuthology.orchestra.run.smithi191.stdout:(8/33): librgw2-19.3.0-4775.g7f24e589.el9.x86_6 18 MB/s | 5.5 MB 00:00 2024-09-14T05:30:18.823 INFO:teuthology.orchestra.run.smithi191.stdout:(9/33): python3-cephfs-19.3.0-4775.g7f24e589.el 1.1 MB/s | 161 kB 00:00 2024-09-14T05:30:18.861 INFO:teuthology.orchestra.run.smithi191.stdout:(10/33): python3-rados-19.3.0-4775.g7f24e589.el 5.4 MB/s | 321 kB 00:00 2024-09-14T05:30:19.066 INFO:teuthology.orchestra.run.smithi191.stdout:(11/33): ceph-common-19.3.0-4775.g7f24e589.el9. 16 MB/s | 22 MB 00:01 2024-09-14T05:30:19.091 INFO:teuthology.orchestra.run.smithi191.stdout:(12/33): python3-rgw-19.3.0-4775.g7f24e589.el9. 425 kB/s | 99 kB 00:00 2024-09-14T05:30:19.117 INFO:teuthology.orchestra.run.smithi191.stdout:(13/33): python3-rbd-19.3.0-4775.g7f24e589.el9. 1.0 MB/s | 306 kB 00:00 2024-09-14T05:30:19.575 INFO:teuthology.orchestra.run.smithi191.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 210 kB/s | 107 kB 00:00 2024-09-14T05:30:19.635 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4775.g7f24e589.el9.x86_64 32/35 2024-09-14T05:30:19.635 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /sys 2024-09-14T05:30:19.636 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /proc 2024-09-14T05:30:19.636 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /mnt 2024-09-14T05:30:19.636 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /var/tmp 2024-09-14T05:30:19.636 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /home 2024-09-14T05:30:19.636 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /root 2024-09-14T05:30:19.636 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /tmp 2024-09-14T05:30:19.636 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:30:19.667 INFO:teuthology.orchestra.run.smithi191.stdout:(15/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 291 kB/s | 160 kB 00:00 2024-09-14T05:30:19.692 INFO:teuthology.orchestra.run.smithi191.stdout:(16/33): libnbd-1.20.2-2.el9.x86_64.rpm 284 kB/s | 171 kB 00:00 2024-09-14T05:30:19.718 INFO:teuthology.orchestra.run.smithi191.stdout:(17/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 318 kB/s | 45 kB 00:00 2024-09-14T05:30:19.835 INFO:teuthology.orchestra.run.smithi191.stdout:(18/33): python3-prettytable-0.7.2-27.el9.noarc 362 kB/s | 42 kB 00:00 2024-09-14T05:30:19.893 INFO:teuthology.orchestra.run.smithi191.stdout:(19/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.4 MB/s | 292 kB 00:00 2024-09-14T05:30:20.010 INFO:teuthology.orchestra.run.smithi191.stdout:(20/33): qatlib-service-24.02.0-1.el9.x86_64.rp 308 kB/s | 36 kB 00:00 2024-09-14T05:30:20.044 INFO:teuthology.orchestra.run.smithi191.stdout:(21/33): librdkafka-1.6.1-102.el9.x86_64.rpm 1.7 MB/s | 662 kB 00:00 2024-09-14T05:30:20.094 INFO:teuthology.orchestra.run.smithi191.stdout:(22/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 6.0 MB/s | 308 kB 00:00 2024-09-14T05:30:20.128 INFO:teuthology.orchestra.run.smithi191.stdout:(23/33): qatlib-24.02.0-1.el9.x86_64.rpm 754 kB/s | 221 kB 00:00 2024-09-14T05:30:20.145 INFO:teuthology.orchestra.run.smithi191.stdout:(24/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 353 kB/s | 47 kB 00:00 2024-09-14T05:30:20.186 INFO:teuthology.orchestra.run.smithi191.stdout:(25/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 445 kB/s | 26 kB 00:00 2024-09-14T05:30:20.212 INFO:teuthology.orchestra.run.smithi191.stdout:(26/33): liboath-2.6.7-2.el9.x86_64.rpm 821 kB/s | 49 kB 00:00 2024-09-14T05:30:20.237 INFO:teuthology.orchestra.run.smithi191.stdout:(27/33): libunwind-1.6.2-1.el9.x86_64.rpm 1.4 MB/s | 67 kB 00:00 2024-09-14T05:30:20.279 INFO:teuthology.orchestra.run.smithi191.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 4.5 MB/s | 191 kB 00:00 2024-09-14T05:30:20.329 INFO:teuthology.orchestra.run.smithi191.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.0 MB/s | 839 kB 00:00 2024-09-14T05:30:20.405 INFO:teuthology.orchestra.run.smithi191.stdout:(30/33): libarrow-9.0.0-11.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-09-14T05:30:20.455 INFO:teuthology.orchestra.run.smithi191.stdout:(31/33): thrift-0.15.0-2.el9.x86_64.rpm 9.0 MB/s | 1.6 MB 00:00 2024-09-14T05:30:20.630 INFO:teuthology.orchestra.run.smithi191.stdout:(32/33): librados2-19.3.0-4775.g7f24e589.el9.x8 12 MB/s | 3.5 MB 00:00 2024-09-14T05:30:20.665 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-radosgw-2:19.3.0-4775.g7f24e589.el9.x86_64 33/35 2024-09-14T05:30:20.690 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4775.g7f24e589.el9.x86_64 33/35 2024-09-14T05:30:20.690 INFO:teuthology.orchestra.run.smithi143.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T05:30:20.690 INFO:teuthology.orchestra.run.smithi143.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-14T05:30:20.690 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-14T05:30:20.690 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-14T05:30:20.690 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:30:20.692 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-14T05:30:20.697 INFO:teuthology.orchestra.run.smithi191.stdout:(33/33): librbd1-19.3.0-4775.g7f24e589.el9.x86_ 11 MB/s | 3.2 MB 00:00 2024-09-14T05:30:20.702 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-09-14T05:30:20.702 INFO:teuthology.orchestra.run.smithi191.stdout:Total 18 MB/s | 62 MB 00:03 2024-09-14T05:30:20.767 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-14T05:30:20.767 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-14T05:30:20.820 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-09-14T05:30:20.857 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-09-14T05:30:20.857 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-09-14T05:30:21.539 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-09-14T05:30:21.539 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-09-14T05:30:21.986 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-14T05:30:21.987 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-base-2:19.3.0-4775.g7f24e589.el9.x86_64 1/35 2024-09-14T05:30:21.987 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-common-2:19.3.0-4775.g7f24e589.el9.x86_64 2/35 2024-09-14T05:30:21.987 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-radosgw-2:19.3.0-4775.g7f24e589.el9.x86_64 3/35 2024-09-14T05:30:21.987 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-selinux-2:19.3.0-4775.g7f24e589.el9.x86_64 4/35 2024-09-14T05:30:21.987 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libcephfs2-2:19.3.0-4775.g7f24e589.el9.x86_64 5/35 2024-09-14T05:30:21.987 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libradosstriper1-2:19.3.0-4775.g7f24e589.el9.x86_6 6/35 2024-09-14T05:30:21.987 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librgw2-2:19.3.0-4775.g7f24e589.el9.x86_64 7/35 2024-09-14T05:30:21.987 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4775.g7f24e589.el9. 8/35 2024-09-14T05:30:21.987 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-ceph-common-2:19.3.0-4775.g7f24e589.el9.x8 9/35 2024-09-14T05:30:21.987 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-cephfs-2:19.3.0-4775.g7f24e589.el9.x86_64 10/35 2024-09-14T05:30:21.988 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-rados-2:19.3.0-4775.g7f24e589.el9.x86_64 11/35 2024-09-14T05:30:21.988 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-rbd-2:19.3.0-4775.g7f24e589.el9.x86_64 12/35 2024-09-14T05:30:21.988 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-rgw-2:19.3.0-4775.g7f24e589.el9.x86_64 13/35 2024-09-14T05:30:21.988 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-14T05:30:21.988 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-14T05:30:21.988 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-14T05:30:21.988 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-14T05:30:21.988 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-14T05:30:21.988 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-14T05:30:21.988 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-14T05:30:21.988 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-14T05:30:21.988 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-14T05:30:21.989 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-14T05:30:21.989 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-14T05:30:21.989 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-14T05:30:21.989 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-14T05:30:21.989 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-14T05:30:21.989 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-14T05:30:21.989 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-14T05:30:21.989 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-14T05:30:21.989 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-14T05:30:21.989 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librados2-2:19.3.0-4775.g7f24e589.el9.x86_64 32/35 2024-09-14T05:30:21.989 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-14T05:30:21.990 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librbd1-2:19.3.0-4775.g7f24e589.el9.x86_64 34/35 2024-09-14T05:30:23.102 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-09-14T05:30:23.108 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-14T05:30:23.108 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:30:23.108 INFO:teuthology.orchestra.run.smithi143.stdout:Upgraded: 2024-09-14T05:30:23.109 INFO:teuthology.orchestra.run.smithi143.stdout: librados2-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T05:30:23.109 INFO:teuthology.orchestra.run.smithi143.stdout: librbd1-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T05:30:23.109 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-09-14T05:30:23.109 INFO:teuthology.orchestra.run.smithi143.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-14T05:30:23.109 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-base-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T05:30:23.109 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-common-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T05:30:23.109 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-radosgw-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T05:30:23.109 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-selinux-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T05:30:23.109 INFO:teuthology.orchestra.run.smithi143.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-14T05:30:23.109 INFO:teuthology.orchestra.run.smithi143.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-14T05:30:23.109 INFO:teuthology.orchestra.run.smithi143.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-14T05:30:23.109 INFO:teuthology.orchestra.run.smithi143.stdout: libcephfs2-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T05:30:23.110 INFO:teuthology.orchestra.run.smithi143.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-14T05:30:23.110 INFO:teuthology.orchestra.run.smithi143.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-14T05:30:23.110 INFO:teuthology.orchestra.run.smithi143.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-14T05:30:23.110 INFO:teuthology.orchestra.run.smithi143.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-14T05:30:23.110 INFO:teuthology.orchestra.run.smithi143.stdout: libradosstriper1-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T05:30:23.110 INFO:teuthology.orchestra.run.smithi143.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-14T05:30:23.110 INFO:teuthology.orchestra.run.smithi143.stdout: librgw2-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T05:30:23.110 INFO:teuthology.orchestra.run.smithi143.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-14T05:30:23.110 INFO:teuthology.orchestra.run.smithi143.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-14T05:30:23.110 INFO:teuthology.orchestra.run.smithi143.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-14T05:30:23.110 INFO:teuthology.orchestra.run.smithi143.stdout: python3-ceph-argparse-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T05:30:23.110 INFO:teuthology.orchestra.run.smithi143.stdout: python3-ceph-common-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T05:30:23.110 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cephfs-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T05:30:23.111 INFO:teuthology.orchestra.run.smithi143.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-14T05:30:23.111 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rados-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T05:30:23.111 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rbd-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T05:30:23.111 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rgw-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T05:30:23.111 INFO:teuthology.orchestra.run.smithi143.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-14T05:30:23.111 INFO:teuthology.orchestra.run.smithi143.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-14T05:30:23.111 INFO:teuthology.orchestra.run.smithi143.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-14T05:30:23.111 INFO:teuthology.orchestra.run.smithi143.stdout: re2-1:20211101-20.el9.x86_64 2024-09-14T05:30:23.111 INFO:teuthology.orchestra.run.smithi143.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-14T05:30:23.111 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:30:23.111 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-09-14T05:30:23.189 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-14T05:30:23.218 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/35 2024-09-14T05:30:23.244 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/35 2024-09-14T05:30:23.266 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-14T05:30:23.291 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-14T05:30:23.460 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-14T05:30:23.500 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph-test 2024-09-14T05:30:23.513 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-14T05:30:23.845 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/35 2024-09-14T05:30:23.953 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : librados2-2:19.3.0-4775.g7f24e589.el9.x86_64 7/35 2024-09-14T05:30:24.046 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librados2-2:19.3.0-4775.g7f24e589.el9.x86_64 7/35 2024-09-14T05:30:24.070 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:00:29 ago on Sat 14 Sep 2024 05:29:55 AM UTC. 2024-09-14T05:30:24.132 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rados-2:19.3.0-4775.g7f24e589.el9.x86_64 8/35 2024-09-14T05:30:24.155 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephfs2-2:19.3.0-4775.g7f24e589.el9.x86_64 9/35 2024-09-14T05:30:24.214 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-09-14T05:30:24.215 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-09-14T05:30:24.215 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repository Size 2024-09-14T05:30:24.215 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-09-14T05:30:24.215 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-09-14T05:30:24.215 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-test x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 50 M 2024-09-14T05:30:24.215 INFO:teuthology.orchestra.run.smithi143.stdout:Installing dependencies: 2024-09-14T05:30:24.215 INFO:teuthology.orchestra.run.smithi143.stdout: libcephsqlite x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 162 k 2024-09-14T05:30:24.215 INFO:teuthology.orchestra.run.smithi143.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-14T05:30:24.215 INFO:teuthology.orchestra.run.smithi143.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-14T05:30:24.215 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:30:24.215 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-09-14T05:30:24.215 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-09-14T05:30:24.216 INFO:teuthology.orchestra.run.smithi143.stdout:Install 4 Packages 2024-09-14T05:30:24.216 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:30:24.216 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 51 M 2024-09-14T05:30:24.216 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 217 M 2024-09-14T05:30:24.216 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-09-14T05:30:24.217 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libcephfs2-2:19.3.0-4775.g7f24e589.el9.x86_64 9/35 2024-09-14T05:30:24.246 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/35 2024-09-14T05:30:24.290 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-ceph-argparse-2:19.3.0-4775.g7f24e589.el9. 11/35 2024-09-14T05:30:24.366 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cephfs-2:19.3.0-4775.g7f24e589.el9.x86_64 12/35 2024-09-14T05:30:24.399 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libradosstriper1-2:19.3.0-4775.g7f24e589.el9.x86_6 13/35 2024-09-14T05:30:24.454 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4775.g7f24e589.el9.x86_6 13/35 2024-09-14T05:30:24.473 INFO:teuthology.orchestra.run.smithi143.stdout:(1/4): libcephsqlite-19.3.0-4775.g7f24e589.el9. 771 kB/s | 162 kB 00:00 2024-09-14T05:30:24.489 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/35 2024-09-14T05:30:24.603 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/35 2024-09-14T05:30:24.648 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/35 2024-09-14T05:30:24.697 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/35 2024-09-14T05:30:24.730 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/35 2024-09-14T05:30:24.758 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/35 2024-09-14T05:30:24.800 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-14T05:30:24.837 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/35 2024-09-14T05:30:25.081 INFO:teuthology.orchestra.run.smithi143.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 105 kB/s | 64 kB 00:00 2024-09-14T05:30:25.106 INFO:teuthology.orchestra.run.smithi143.stdout:(3/4): socat-1.7.4.1-6.el9.x86_64.rpm 361 kB/s | 304 kB 00:00 2024-09-14T05:30:25.155 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-14T05:30:25.197 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : librbd1-2:19.3.0-4775.g7f24e589.el9.x86_64 23/35 2024-09-14T05:30:25.274 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librbd1-2:19.3.0-4775.g7f24e589.el9.x86_64 23/35 2024-09-14T05:30:25.311 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rbd-2:19.3.0-4775.g7f24e589.el9.x86_64 24/35 2024-09-14T05:30:25.354 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-14T05:30:25.823 INFO:teuthology.orchestra.run.smithi143.stdout:(4/4): ceph-test-19.3.0-4775.g7f24e589.el9.x86_ 32 MB/s | 50 MB 00:01 2024-09-14T05:30:25.826 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-09-14T05:30:25.826 INFO:teuthology.orchestra.run.smithi143.stdout:Total 32 MB/s | 51 MB 00:01 2024-09-14T05:30:25.853 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-09-14T05:30:25.875 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-09-14T05:30:25.875 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-09-14T05:30:25.882 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-14T05:30:25.916 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librgw2-2:19.3.0-4775.g7f24e589.el9.x86_64 27/35 2024-09-14T05:30:26.005 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librgw2-2:19.3.0-4775.g7f24e589.el9.x86_64 27/35 2024-09-14T05:30:26.084 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rgw-2:19.3.0-4775.g7f24e589.el9.x86_64 28/35 2024-09-14T05:30:26.110 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-ceph-common-2:19.3.0-4775.g7f24e589.el9.x8 29/35 2024-09-14T05:30:26.273 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-09-14T05:30:26.523 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-09-14T05:30:26.815 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-09-14T05:30:26.892 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-14T05:30:26.940 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-14T05:30:26.967 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libcephsqlite-2:19.3.0-4775.g7f24e589.el9.x86_64 3/4 2024-09-14T05:30:28.432 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-common-2:19.3.0-4775.g7f24e589.el9.x86_64 30/35 2024-09-14T05:30:28.465 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-common-2:19.3.0-4775.g7f24e589.el9.x86_64 30/35 2024-09-14T05:30:29.056 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-common-2:19.3.0-4775.g7f24e589.el9.x86_64 30/35 2024-09-14T05:30:29.101 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-base-2:19.3.0-4775.g7f24e589.el9.x86_64 31/35 2024-09-14T05:30:29.174 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-base-2:19.3.0-4775.g7f24e589.el9.x86_64 31/35 2024-09-14T05:30:29.174 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-14T05:30:29.174 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-14T05:30:29.175 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:30:29.211 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-selinux-2:19.3.0-4775.g7f24e589.el9.x86_64 32/35 2024-09-14T05:30:31.721 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4775.g7f24e589.el9.x86_64 3/4 2024-09-14T05:30:31.807 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-test-2:19.3.0-4775.g7f24e589.el9.x86_64 4/4 2024-09-14T05:30:34.276 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-test-2:19.3.0-4775.g7f24e589.el9.x86_64 4/4 2024-09-14T05:30:34.277 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-test-2:19.3.0-4775.g7f24e589.el9.x86_64 1/4 2024-09-14T05:30:34.277 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libcephsqlite-2:19.3.0-4775.g7f24e589.el9.x86_64 2/4 2024-09-14T05:30:34.277 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-14T05:30:34.599 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-14T05:30:34.599 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:30:34.599 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-09-14T05:30:34.599 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-test-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T05:30:34.599 INFO:teuthology.orchestra.run.smithi143.stdout: libcephsqlite-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T05:30:34.599 INFO:teuthology.orchestra.run.smithi143.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-14T05:30:34.599 INFO:teuthology.orchestra.run.smithi143.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-14T05:30:34.599 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:30:34.599 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-09-14T05:30:34.931 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph 2024-09-14T05:30:35.509 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:00:40 ago on Sat 14 Sep 2024 05:29:55 AM UTC. 2024-09-14T05:30:35.654 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-09-14T05:30:35.657 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-09-14T05:30:35.657 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repository Size 2024-09-14T05:30:35.657 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-09-14T05:30:35.657 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-09-14T05:30:35.657 INFO:teuthology.orchestra.run.smithi143.stdout: ceph x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 6.6 k 2024-09-14T05:30:35.657 INFO:teuthology.orchestra.run.smithi143.stdout:Installing dependencies: 2024-09-14T05:30:35.657 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mds x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 2.4 M 2024-09-14T05:30:35.657 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 1.5 M 2024-09-14T05:30:35.657 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4775.g7f24e589.el9 ceph-noarch 291 k 2024-09-14T05:30:35.657 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mon x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 4.8 M 2024-09-14T05:30:35.657 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-osd x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 17 M 2024-09-14T05:30:35.658 INFO:teuthology.orchestra.run.smithi143.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-14T05:30:35.658 INFO:teuthology.orchestra.run.smithi143.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-14T05:30:35.658 INFO:teuthology.orchestra.run.smithi143.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-14T05:30:35.658 INFO:teuthology.orchestra.run.smithi143.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-14T05:30:35.658 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-14T05:30:35.658 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-14T05:30:35.658 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-14T05:30:35.658 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-14T05:30:35.658 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-14T05:30:35.658 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-14T05:30:35.658 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-collections 2024-09-14T05:30:35.658 INFO:teuthology.orchestra.run.smithi143.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-14T05:30:35.658 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-14T05:30:35.659 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-14T05:30:35.659 INFO:teuthology.orchestra.run.smithi143.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-14T05:30:35.659 INFO:teuthology.orchestra.run.smithi143.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-14T05:30:35.659 INFO:teuthology.orchestra.run.smithi143.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-14T05:30:35.659 INFO:teuthology.orchestra.run.smithi143.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-14T05:30:35.659 INFO:teuthology.orchestra.run.smithi143.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-14T05:30:35.659 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-14T05:30:35.659 INFO:teuthology.orchestra.run.smithi143.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-14T05:30:35.659 INFO:teuthology.orchestra.run.smithi143.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-14T05:30:35.659 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-14T05:30:35.659 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-14T05:30:35.659 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-14T05:30:35.659 INFO:teuthology.orchestra.run.smithi143.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-14T05:30:35.660 INFO:teuthology.orchestra.run.smithi143.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-14T05:30:35.660 INFO:teuthology.orchestra.run.smithi143.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-14T05:30:35.660 INFO:teuthology.orchestra.run.smithi143.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-14T05:30:35.660 INFO:teuthology.orchestra.run.smithi143.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-14T05:30:35.660 INFO:teuthology.orchestra.run.smithi143.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-14T05:30:35.660 INFO:teuthology.orchestra.run.smithi143.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-14T05:30:35.660 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:30:35.660 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-09-14T05:30:35.660 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-09-14T05:30:35.660 INFO:teuthology.orchestra.run.smithi143.stdout:Install 37 Packages 2024-09-14T05:30:35.660 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:30:35.661 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 31 M 2024-09-14T05:30:35.661 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 109 M 2024-09-14T05:30:35.661 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-09-14T05:30:35.926 INFO:teuthology.orchestra.run.smithi143.stdout:(1/37): ceph-19.3.0-4775.g7f24e589.el9.x86_64.r 53 kB/s | 6.6 kB 00:00 2024-09-14T05:30:36.109 INFO:teuthology.orchestra.run.smithi143.stdout:(2/37): ceph-mgr-19.3.0-4775.g7f24e589.el9.x86_ 5.0 MB/s | 1.5 MB 00:00 2024-09-14T05:30:36.159 INFO:teuthology.orchestra.run.smithi143.stdout:(3/37): ceph-mds-19.3.0-4775.g7f24e589.el9.x86_ 6.7 MB/s | 2.4 MB 00:00 2024-09-14T05:30:36.251 INFO:teuthology.orchestra.run.smithi143.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4775.g7f24 3.1 MB/s | 291 kB 00:00 2024-09-14T05:30:36.377 INFO:teuthology.orchestra.run.smithi143.stdout:(5/37): ceph-mon-19.3.0-4775.g7f24e589.el9.x86_ 11 MB/s | 4.8 MB 00:00 2024-09-14T05:30:36.502 INFO:teuthology.orchestra.run.smithi143.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.0 MB/s | 253 kB 00:00 2024-09-14T05:30:36.703 INFO:teuthology.orchestra.run.smithi143.stdout:(7/37): ceph-osd-19.3.0-4775.g7f24e589.el9.x86_ 29 MB/s | 17 MB 00:00 2024-09-14T05:30:36.728 INFO:teuthology.orchestra.run.smithi143.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 473 kB/s | 106 kB 00:00 2024-09-14T05:30:36.803 INFO:teuthology.orchestra.run.smithi143.stdout:(9/37): python3-requests-2.25.1-8.el9.noarch.rp 1.6 MB/s | 125 kB 00:00 2024-09-14T05:30:36.845 INFO:teuthology.orchestra.run.smithi143.stdout:(10/37): python3-cryptography-36.0.1-4.el9.x86_ 2.6 MB/s | 1.2 MB 00:00 2024-09-14T05:30:36.870 INFO:teuthology.orchestra.run.smithi143.stdout:(11/37): python3-pycparser-2.20-6.el9.noarch.rp 807 kB/s | 135 kB 00:00 2024-09-14T05:30:36.896 INFO:teuthology.orchestra.run.smithi143.stdout:(12/37): python3-urllib3-1.26.5-6.el9.noarch.rp 2.3 MB/s | 215 kB 00:00 2024-09-14T05:30:37.438 INFO:teuthology.orchestra.run.smithi143.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 64 kB/s | 35 kB 00:00 2024-09-14T05:30:37.613 INFO:teuthology.orchestra.run.smithi143.stdout:(14/37): lua-5.4.4-4.el9.x86_64.rpm 245 kB/s | 188 kB 00:00 2024-09-14T05:30:37.647 INFO:teuthology.orchestra.run.smithi143.stdout:(15/37): python3-mako-1.1.4-6.el9.noarch.rpm 222 kB/s | 172 kB 00:00 2024-09-14T05:30:38.064 INFO:teuthology.orchestra.run.smithi143.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 123 kB/s | 77 kB 00:00 2024-09-14T05:30:38.222 INFO:teuthology.orchestra.run.smithi143.stdout:(17/37): lua-devel-5.4.4-4.el9.x86_64.rpm 141 kB/s | 22 kB 00:00 2024-09-14T05:30:38.247 INFO:teuthology.orchestra.run.smithi143.stdout:(18/37): python3-pytz-2021.1-5.el9.noarch.rpm 80 kB/s | 51 kB 00:00 2024-09-14T05:30:38.264 INFO:teuthology.orchestra.run.smithi143.stdout:(19/37): python3-toml-0.10.2-6.el9.noarch.rpm 67 kB/s | 42 kB 00:00 2024-09-14T05:30:38.323 INFO:teuthology.orchestra.run.smithi143.stdout:(20/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 747 kB/s | 43 kB 00:00 2024-09-14T05:30:38.348 INFO:teuthology.orchestra.run.smithi143.stdout:(21/37): luarocks-3.9.2-5.el9.noarch.rpm 1.2 MB/s | 151 kB 00:00 2024-09-14T05:30:38.373 INFO:teuthology.orchestra.run.smithi143.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 429 kB/s | 11 kB 00:00 2024-09-14T05:30:38.399 INFO:teuthology.orchestra.run.smithi143.stdout:(23/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-09-14T05:30:38.424 INFO:teuthology.orchestra.run.smithi143.stdout:(24/37): python3-jaraco-classes-3.2.1-5.el9.noa 353 kB/s | 18 kB 00:00 2024-09-14T05:30:38.474 INFO:teuthology.orchestra.run.smithi143.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 308 kB/s | 23 kB 00:00 2024-09-14T05:30:38.499 INFO:teuthology.orchestra.run.smithi143.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 259 kB/s | 19 kB 00:00 2024-09-14T05:30:38.525 INFO:teuthology.orchestra.run.smithi143.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-09-14T05:30:38.550 INFO:teuthology.orchestra.run.smithi143.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 919 kB/s | 46 kB 00:00 2024-09-14T05:30:38.575 INFO:teuthology.orchestra.run.smithi143.stdout:(29/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 MB/s | 358 kB 00:00 2024-09-14T05:30:38.600 INFO:teuthology.orchestra.run.smithi143.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-09-14T05:30:38.626 INFO:teuthology.orchestra.run.smithi143.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 326 kB/s | 16 kB 00:00 2024-09-14T05:30:38.655 INFO:teuthology.orchestra.run.smithi143.stdout:(32/37): python3-pecan-1.4.2-3.el9.noarch.rpm 2.5 MB/s | 272 kB 00:00 2024-09-14T05:30:38.672 INFO:teuthology.orchestra.run.smithi143.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 785 kB/s | 36 kB 00:00 2024-09-14T05:30:38.697 INFO:teuthology.orchestra.run.smithi143.stdout:(34/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 934 kB/s | 90 kB 00:00 2024-09-14T05:30:38.714 INFO:teuthology.orchestra.run.smithi143.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 1.2 MB/s | 20 kB 00:00 2024-09-14T05:30:38.739 INFO:teuthology.orchestra.run.smithi143.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-09-14T05:30:38.765 INFO:teuthology.orchestra.run.smithi143.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.5 MB/s | 427 kB 00:00 2024-09-14T05:30:38.770 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-09-14T05:30:38.770 INFO:teuthology.orchestra.run.smithi143.stdout:Total 9.9 MB/s | 31 MB 00:03 2024-09-14T05:30:38.923 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-09-14T05:30:38.963 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-09-14T05:30:38.963 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-09-14T05:30:39.426 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-09-14T05:30:39.426 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-09-14T05:30:39.514 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4775.g7f24e589.el9.x86_64 32/35 2024-09-14T05:30:39.514 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /sys 2024-09-14T05:30:39.514 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /proc 2024-09-14T05:30:39.514 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /mnt 2024-09-14T05:30:39.514 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /var/tmp 2024-09-14T05:30:39.514 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /home 2024-09-14T05:30:39.514 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /root 2024-09-14T05:30:39.514 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /tmp 2024-09-14T05:30:39.514 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:30:40.537 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-radosgw-2:19.3.0-4775.g7f24e589.el9.x86_64 33/35 2024-09-14T05:30:40.562 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4775.g7f24e589.el9.x86_64 33/35 2024-09-14T05:30:40.562 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T05:30:40.562 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-14T05:30:40.562 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-14T05:30:40.562 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-14T05:30:40.563 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:30:40.564 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-14T05:30:40.664 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-14T05:30:40.664 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-14T05:30:40.788 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-09-14T05:30:40.843 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-14T05:30:40.875 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-14T05:30:40.895 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-14T05:30:40.965 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-14T05:30:41.002 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-14T05:30:41.035 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-14T05:30:41.126 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-14T05:30:41.213 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-14T05:30:41.254 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-14T05:30:41.290 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-14T05:30:41.320 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-14T05:30:41.354 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-14T05:30:41.387 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-14T05:30:41.428 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-14T05:30:41.463 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-14T05:30:41.498 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-14T05:30:41.546 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-14T05:30:41.578 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-14T05:30:41.653 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-14T05:30:41.798 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-14T05:30:41.894 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-14T05:30:41.947 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-14T05:30:41.978 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-14T05:30:41.978 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-base-2:19.3.0-4775.g7f24e589.el9.x86_64 1/35 2024-09-14T05:30:41.978 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-common-2:19.3.0-4775.g7f24e589.el9.x86_64 2/35 2024-09-14T05:30:41.979 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-radosgw-2:19.3.0-4775.g7f24e589.el9.x86_64 3/35 2024-09-14T05:30:41.979 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-selinux-2:19.3.0-4775.g7f24e589.el9.x86_64 4/35 2024-09-14T05:30:41.979 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephfs2-2:19.3.0-4775.g7f24e589.el9.x86_64 5/35 2024-09-14T05:30:41.979 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libradosstriper1-2:19.3.0-4775.g7f24e589.el9.x86_6 6/35 2024-09-14T05:30:41.979 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librgw2-2:19.3.0-4775.g7f24e589.el9.x86_64 7/35 2024-09-14T05:30:41.979 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4775.g7f24e589.el9. 8/35 2024-09-14T05:30:41.979 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-ceph-common-2:19.3.0-4775.g7f24e589.el9.x8 9/35 2024-09-14T05:30:41.979 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cephfs-2:19.3.0-4775.g7f24e589.el9.x86_64 10/35 2024-09-14T05:30:41.979 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rados-2:19.3.0-4775.g7f24e589.el9.x86_64 11/35 2024-09-14T05:30:41.979 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rbd-2:19.3.0-4775.g7f24e589.el9.x86_64 12/35 2024-09-14T05:30:41.979 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rgw-2:19.3.0-4775.g7f24e589.el9.x86_64 13/35 2024-09-14T05:30:41.979 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-14T05:30:41.979 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-14T05:30:41.980 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-14T05:30:41.980 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-14T05:30:41.980 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-14T05:30:41.980 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-14T05:30:41.980 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-14T05:30:41.980 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-14T05:30:41.980 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-14T05:30:41.980 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-14T05:30:41.980 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-14T05:30:41.980 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-14T05:30:41.980 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-14T05:30:41.980 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-14T05:30:41.981 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-14T05:30:41.981 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-14T05:30:41.982 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-14T05:30:41.982 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-14T05:30:41.982 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados2-2:19.3.0-4775.g7f24e589.el9.x86_64 32/35 2024-09-14T05:30:41.982 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-14T05:30:41.982 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librbd1-2:19.3.0-4775.g7f24e589.el9.x86_64 34/35 2024-09-14T05:30:41.986 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-14T05:30:42.039 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-14T05:30:42.094 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-14T05:30:42.278 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-14T05:30:42.320 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-14T05:30:42.397 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-14T05:30:42.510 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-14T05:30:42.552 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-14T05:30:42.678 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-14T05:30:42.852 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4775.g7f24e589.el9. 32/37 2024-09-14T05:30:42.879 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mgr-2:19.3.0-4775.g7f24e589.el9.x86_64 33/37 2024-09-14T05:30:42.906 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4775.g7f24e589.el9.x86_64 33/37 2024-09-14T05:30:42.906 INFO:teuthology.orchestra.run.smithi143.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T05:30:42.907 INFO:teuthology.orchestra.run.smithi143.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-14T05:30:42.907 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-14T05:30:42.907 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-14T05:30:42.907 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:30:42.938 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-14T05:30:42.938 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:30:42.938 INFO:teuthology.orchestra.run.smithi191.stdout:Upgraded: 2024-09-14T05:30:42.938 INFO:teuthology.orchestra.run.smithi191.stdout: librados2-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T05:30:42.938 INFO:teuthology.orchestra.run.smithi191.stdout: librbd1-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T05:30:42.938 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-09-14T05:30:42.939 INFO:teuthology.orchestra.run.smithi191.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-14T05:30:42.939 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-base-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T05:30:42.939 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-common-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T05:30:42.939 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-radosgw-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T05:30:42.939 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-selinux-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T05:30:42.939 INFO:teuthology.orchestra.run.smithi191.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-14T05:30:42.939 INFO:teuthology.orchestra.run.smithi191.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-14T05:30:42.939 INFO:teuthology.orchestra.run.smithi191.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-14T05:30:42.939 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs2-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T05:30:42.939 INFO:teuthology.orchestra.run.smithi191.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-14T05:30:42.939 INFO:teuthology.orchestra.run.smithi191.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-14T05:30:42.939 INFO:teuthology.orchestra.run.smithi191.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-14T05:30:42.940 INFO:teuthology.orchestra.run.smithi191.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-14T05:30:42.940 INFO:teuthology.orchestra.run.smithi191.stdout: libradosstriper1-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T05:30:42.940 INFO:teuthology.orchestra.run.smithi191.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-14T05:30:42.940 INFO:teuthology.orchestra.run.smithi191.stdout: librgw2-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T05:30:42.940 INFO:teuthology.orchestra.run.smithi191.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-14T05:30:42.940 INFO:teuthology.orchestra.run.smithi191.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-14T05:30:42.940 INFO:teuthology.orchestra.run.smithi191.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-14T05:30:42.940 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-argparse-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T05:30:42.940 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-common-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T05:30:42.940 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cephfs-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T05:30:42.941 INFO:teuthology.orchestra.run.smithi191.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-14T05:30:42.941 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rados-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T05:30:42.941 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rbd-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T05:30:42.941 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rgw-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T05:30:42.941 INFO:teuthology.orchestra.run.smithi191.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-14T05:30:42.941 INFO:teuthology.orchestra.run.smithi191.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-14T05:30:42.941 INFO:teuthology.orchestra.run.smithi191.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-14T05:30:42.941 INFO:teuthology.orchestra.run.smithi191.stdout: re2-1:20211101-20.el9.x86_64 2024-09-14T05:30:42.941 INFO:teuthology.orchestra.run.smithi191.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-14T05:30:42.941 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:30:42.942 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-14T05:30:43.325 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-test 2024-09-14T05:30:43.896 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:29 ago on Sat 14 Sep 2024 05:30:14 AM UTC. 2024-09-14T05:30:44.029 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-14T05:30:44.030 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-14T05:30:44.030 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-09-14T05:30:44.030 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-14T05:30:44.030 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-09-14T05:30:44.030 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-test x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 50 M 2024-09-14T05:30:44.030 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-09-14T05:30:44.030 INFO:teuthology.orchestra.run.smithi191.stdout: libcephsqlite x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 162 k 2024-09-14T05:30:44.031 INFO:teuthology.orchestra.run.smithi191.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-14T05:30:44.031 INFO:teuthology.orchestra.run.smithi191.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-14T05:30:44.031 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:30:44.031 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-09-14T05:30:44.031 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-14T05:30:44.031 INFO:teuthology.orchestra.run.smithi191.stdout:Install 4 Packages 2024-09-14T05:30:44.031 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:30:44.031 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 51 M 2024-09-14T05:30:44.031 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 217 M 2024-09-14T05:30:44.032 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-09-14T05:30:44.261 INFO:teuthology.orchestra.run.smithi191.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.4 MB/s | 304 kB 00:00 2024-09-14T05:30:44.286 INFO:teuthology.orchestra.run.smithi191.stdout:(2/4): libcephsqlite-19.3.0-4775.g7f24e589.el9. 693 kB/s | 162 kB 00:00 2024-09-14T05:30:44.303 INFO:teuthology.orchestra.run.smithi191.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-14T05:30:44.364 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-osd-2:19.3.0-4775.g7f24e589.el9.x86_64 34/37 2024-09-14T05:30:44.391 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-osd-2:19.3.0-4775.g7f24e589.el9.x86_64 34/37 2024-09-14T05:30:44.391 INFO:teuthology.orchestra.run.smithi143.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T05:30:44.391 INFO:teuthology.orchestra.run.smithi143.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-14T05:30:44.391 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-14T05:30:44.392 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-14T05:30:44.392 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:30:44.840 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mon-2:19.3.0-4775.g7f24e589.el9.x86_64 35/37 2024-09-14T05:30:44.864 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mon-2:19.3.0-4775.g7f24e589.el9.x86_64 35/37 2024-09-14T05:30:44.864 INFO:teuthology.orchestra.run.smithi143.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T05:30:44.864 INFO:teuthology.orchestra.run.smithi143.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-14T05:30:44.864 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-14T05:30:44.864 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-14T05:30:44.864 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:30:45.098 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mds-2:19.3.0-4775.g7f24e589.el9.x86_64 36/37 2024-09-14T05:30:45.122 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mds-2:19.3.0-4775.g7f24e589.el9.x86_64 36/37 2024-09-14T05:30:45.122 INFO:teuthology.orchestra.run.smithi143.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T05:30:45.122 INFO:teuthology.orchestra.run.smithi143.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-14T05:30:45.122 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-14T05:30:45.123 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-14T05:30:45.123 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:30:45.179 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-2:19.3.0-4775.g7f24e589.el9.x86_64 37/37 2024-09-14T05:30:45.637 INFO:teuthology.orchestra.run.smithi191.stdout:(4/4): ceph-test-19.3.0-4775.g7f24e589.el9.x86_ 32 MB/s | 50 MB 00:01 2024-09-14T05:30:45.639 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-09-14T05:30:45.639 INFO:teuthology.orchestra.run.smithi191.stdout:Total 32 MB/s | 51 MB 00:01 2024-09-14T05:30:45.667 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-09-14T05:30:45.687 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-09-14T05:30:45.687 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-09-14T05:30:46.083 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-09-14T05:30:46.083 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-09-14T05:30:46.627 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-09-14T05:30:46.717 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-14T05:30:46.769 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-14T05:30:46.805 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephsqlite-2:19.3.0-4775.g7f24e589.el9.x86_64 3/4 2024-09-14T05:30:46.986 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-2:19.3.0-4775.g7f24e589.el9.x86_64 37/37 2024-09-14T05:30:46.986 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-2:19.3.0-4775.g7f24e589.el9.x86_64 1/37 2024-09-14T05:30:46.986 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mds-2:19.3.0-4775.g7f24e589.el9.x86_64 2/37 2024-09-14T05:30:46.986 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mgr-2:19.3.0-4775.g7f24e589.el9.x86_64 3/37 2024-09-14T05:30:46.986 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mon-2:19.3.0-4775.g7f24e589.el9.x86_64 4/37 2024-09-14T05:30:46.987 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-osd-2:19.3.0-4775.g7f24e589.el9.x86_64 5/37 2024-09-14T05:30:46.987 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4775.g7f24e589.el9. 6/37 2024-09-14T05:30:46.987 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-14T05:30:46.987 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-14T05:30:46.987 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-14T05:30:46.987 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-14T05:30:46.987 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-14T05:30:46.987 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-14T05:30:46.987 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-14T05:30:46.987 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-14T05:30:46.987 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-14T05:30:46.987 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-14T05:30:46.987 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-14T05:30:46.987 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-14T05:30:46.988 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-14T05:30:46.988 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-14T05:30:46.988 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-14T05:30:46.988 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-14T05:30:46.988 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-14T05:30:46.988 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-14T05:30:46.988 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-14T05:30:46.988 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-14T05:30:46.988 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-14T05:30:46.988 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-14T05:30:46.988 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-14T05:30:46.988 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-14T05:30:46.989 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-14T05:30:46.989 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-14T05:30:46.989 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-14T05:30:46.989 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-14T05:30:46.989 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-14T05:30:46.989 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-14T05:30:47.939 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-14T05:30:47.940 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:30:47.940 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-09-14T05:30:47.940 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T05:30:47.940 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mds-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T05:30:47.940 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T05:30:47.940 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-modules-core-2:19.3.0-4775.g7f24e589.el9.noarch 2024-09-14T05:30:47.940 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mon-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T05:30:47.940 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-osd-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T05:30:47.940 INFO:teuthology.orchestra.run.smithi143.stdout: lua-5.4.4-4.el9.x86_64 2024-09-14T05:30:47.941 INFO:teuthology.orchestra.run.smithi143.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-14T05:30:47.941 INFO:teuthology.orchestra.run.smithi143.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-14T05:30:47.941 INFO:teuthology.orchestra.run.smithi143.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-14T05:30:47.941 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-14T05:30:47.941 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-14T05:30:47.941 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-14T05:30:47.941 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-14T05:30:47.941 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-14T05:30:47.941 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-14T05:30:47.941 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-14T05:30:47.941 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-14T05:30:47.941 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-14T05:30:47.942 INFO:teuthology.orchestra.run.smithi143.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-14T05:30:47.942 INFO:teuthology.orchestra.run.smithi143.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-14T05:30:47.942 INFO:teuthology.orchestra.run.smithi143.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-14T05:30:47.942 INFO:teuthology.orchestra.run.smithi143.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-14T05:30:47.942 INFO:teuthology.orchestra.run.smithi143.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-14T05:30:47.942 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-14T05:30:47.942 INFO:teuthology.orchestra.run.smithi143.stdout: python3-ply-3.11-14.el9.noarch 2024-09-14T05:30:47.942 INFO:teuthology.orchestra.run.smithi143.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-14T05:30:47.942 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-14T05:30:47.942 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-14T05:30:47.942 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-14T05:30:47.943 INFO:teuthology.orchestra.run.smithi143.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-14T05:30:47.943 INFO:teuthology.orchestra.run.smithi143.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-14T05:30:47.943 INFO:teuthology.orchestra.run.smithi143.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-14T05:30:47.943 INFO:teuthology.orchestra.run.smithi143.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-14T05:30:47.943 INFO:teuthology.orchestra.run.smithi143.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-14T05:30:47.943 INFO:teuthology.orchestra.run.smithi143.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-14T05:30:47.943 INFO:teuthology.orchestra.run.smithi143.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-14T05:30:47.943 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:30:47.943 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-09-14T05:30:48.307 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph-base 2024-09-14T05:30:48.873 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:00:53 ago on Sat 14 Sep 2024 05:29:55 AM UTC. 2024-09-14T05:30:48.963 INFO:teuthology.orchestra.run.smithi143.stdout:Package ceph-base-2:19.3.0-4775.g7f24e589.el9.x86_64 is already installed. 2024-09-14T05:30:49.004 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-09-14T05:30:49.005 INFO:teuthology.orchestra.run.smithi143.stdout:Nothing to do. 2024-09-14T05:30:49.005 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-09-14T05:30:49.062 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install cephadm 2024-09-14T05:30:49.619 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:00:54 ago on Sat 14 Sep 2024 05:29:55 AM UTC. 2024-09-14T05:30:49.753 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-09-14T05:30:49.754 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-09-14T05:30:49.754 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repository Size 2024-09-14T05:30:49.754 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-09-14T05:30:49.754 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-09-14T05:30:49.754 INFO:teuthology.orchestra.run.smithi143.stdout: cephadm noarch 2:19.3.0-4775.g7f24e589.el9 ceph-noarch 783 k 2024-09-14T05:30:49.754 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:30:49.754 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-09-14T05:30:49.755 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-09-14T05:30:49.755 INFO:teuthology.orchestra.run.smithi143.stdout:Install 1 Package 2024-09-14T05:30:49.755 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:30:49.755 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 783 k 2024-09-14T05:30:49.755 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 789 k 2024-09-14T05:30:49.755 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-09-14T05:30:50.042 INFO:teuthology.orchestra.run.smithi143.stdout:cephadm-19.3.0-4775.g7f24e589.el9.noarch.rpm 2.7 MB/s | 783 kB 00:00 2024-09-14T05:30:50.043 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-09-14T05:30:50.043 INFO:teuthology.orchestra.run.smithi143.stdout:Total 2.6 MB/s | 783 kB 00:00 2024-09-14T05:30:50.043 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-09-14T05:30:50.049 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-09-14T05:30:50.049 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-09-14T05:30:50.055 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-09-14T05:30:50.055 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-09-14T05:30:50.126 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-09-14T05:30:50.550 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: cephadm-2:19.3.0-4775.g7f24e589.el9.noarch 1/1 2024-09-14T05:30:50.637 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : cephadm-2:19.3.0-4775.g7f24e589.el9.noarch 1/1 2024-09-14T05:30:51.106 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: cephadm-2:19.3.0-4775.g7f24e589.el9.noarch 1/1 2024-09-14T05:30:51.360 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : cephadm-2:19.3.0-4775.g7f24e589.el9.noarch 1/1 2024-09-14T05:30:51.360 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:30:51.360 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-09-14T05:30:51.361 INFO:teuthology.orchestra.run.smithi143.stdout: cephadm-2:19.3.0-4775.g7f24e589.el9.noarch 2024-09-14T05:30:51.361 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:30:51.361 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-09-14T05:30:51.555 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph-immutable-object-cache 2024-09-14T05:30:51.560 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4775.g7f24e589.el9.x86_64 3/4 2024-09-14T05:30:51.653 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-test-2:19.3.0-4775.g7f24e589.el9.x86_64 4/4 2024-09-14T05:30:52.128 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:00:57 ago on Sat 14 Sep 2024 05:29:55 AM UTC. 2024-09-14T05:30:52.265 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-09-14T05:30:52.265 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-09-14T05:30:52.265 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repo Size 2024-09-14T05:30:52.265 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-09-14T05:30:52.265 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-09-14T05:30:52.265 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 145 k 2024-09-14T05:30:52.265 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:30:52.265 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-09-14T05:30:52.265 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-09-14T05:30:52.265 INFO:teuthology.orchestra.run.smithi143.stdout:Install 1 Package 2024-09-14T05:30:52.266 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:30:52.266 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 145 k 2024-09-14T05:30:52.266 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 432 k 2024-09-14T05:30:52.266 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-09-14T05:30:52.483 INFO:teuthology.orchestra.run.smithi143.stdout:ceph-immutable-object-cache-19.3.0-4775.g7f24e5 665 kB/s | 145 kB 00:00 2024-09-14T05:30:52.484 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-09-14T05:30:52.484 INFO:teuthology.orchestra.run.smithi143.stdout:Total 661 kB/s | 145 kB 00:00 2024-09-14T05:30:52.484 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-09-14T05:30:52.491 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-09-14T05:30:52.491 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-09-14T05:30:52.548 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-09-14T05:30:52.548 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-09-14T05:30:52.689 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-09-14T05:30:52.775 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4775.g7f24e589. 1/1 2024-09-14T05:30:52.799 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4775.g7f24e589. 1/1 2024-09-14T05:30:52.799 INFO:teuthology.orchestra.run.smithi143.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T05:30:52.799 INFO:teuthology.orchestra.run.smithi143.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-14T05:30:52.799 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:30:53.676 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4775.g7f24e589. 1/1 2024-09-14T05:30:53.676 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:30:53.676 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-09-14T05:30:53.676 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-immutable-object-cache-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T05:30:53.676 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:30:53.676 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-09-14T05:30:53.886 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph-mgr 2024-09-14T05:30:54.279 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-test-2:19.3.0-4775.g7f24e589.el9.x86_64 4/4 2024-09-14T05:30:54.279 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-test-2:19.3.0-4775.g7f24e589.el9.x86_64 1/4 2024-09-14T05:30:54.279 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephsqlite-2:19.3.0-4775.g7f24e589.el9.x86_64 2/4 2024-09-14T05:30:54.279 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-14T05:30:54.460 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:00:59 ago on Sat 14 Sep 2024 05:29:55 AM UTC. 2024-09-14T05:30:54.553 INFO:teuthology.orchestra.run.smithi143.stdout:Package ceph-mgr-2:19.3.0-4775.g7f24e589.el9.x86_64 is already installed. 2024-09-14T05:30:54.598 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-09-14T05:30:54.599 INFO:teuthology.orchestra.run.smithi143.stdout:Nothing to do. 2024-09-14T05:30:54.599 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-09-14T05:30:54.630 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-14T05:30:54.630 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:30:54.630 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-09-14T05:30:54.630 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-test-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T05:30:54.631 INFO:teuthology.orchestra.run.smithi191.stdout: libcephsqlite-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T05:30:54.631 INFO:teuthology.orchestra.run.smithi191.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-14T05:30:54.631 INFO:teuthology.orchestra.run.smithi191.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-14T05:30:54.631 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:30:54.631 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-14T05:30:54.664 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph-mgr-dashboard 2024-09-14T05:30:54.916 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph 2024-09-14T05:30:55.219 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:01:00 ago on Sat 14 Sep 2024 05:29:55 AM UTC. 2024-09-14T05:30:55.351 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-09-14T05:30:55.352 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-09-14T05:30:55.353 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repository Size 2024-09-14T05:30:55.353 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-09-14T05:30:55.353 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-09-14T05:30:55.353 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4775.g7f24e589.el9 ceph-noarch 14 M 2024-09-14T05:30:55.353 INFO:teuthology.orchestra.run.smithi143.stdout:Installing dependencies: 2024-09-14T05:30:55.353 INFO:teuthology.orchestra.run.smithi143.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-14T05:30:55.353 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4775.g7f24e589.el9 ceph-noarch 34 k 2024-09-14T05:30:55.353 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4775.g7f24e589.el9 ceph-noarch 16 k 2024-09-14T05:30:55.353 INFO:teuthology.orchestra.run.smithi143.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-14T05:30:55.353 INFO:teuthology.orchestra.run.smithi143.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-14T05:30:55.354 INFO:teuthology.orchestra.run.smithi143.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-14T05:30:55.354 INFO:teuthology.orchestra.run.smithi143.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-14T05:30:55.354 INFO:teuthology.orchestra.run.smithi143.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-14T05:30:55.354 INFO:teuthology.orchestra.run.smithi143.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-14T05:30:55.354 INFO:teuthology.orchestra.run.smithi143.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-14T05:30:55.354 INFO:teuthology.orchestra.run.smithi143.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-14T05:30:55.354 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:30:55.354 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-09-14T05:30:55.354 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-09-14T05:30:55.354 INFO:teuthology.orchestra.run.smithi143.stdout:Install 12 Packages 2024-09-14T05:30:55.354 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:30:55.355 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 19 M 2024-09-14T05:30:55.355 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 159 M 2024-09-14T05:30:55.355 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-09-14T05:30:55.484 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:41 ago on Sat 14 Sep 2024 05:30:14 AM UTC. 2024-09-14T05:30:55.630 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-14T05:30:55.632 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-14T05:30:55.632 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-09-14T05:30:55.632 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-14T05:30:55.632 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-09-14T05:30:55.632 INFO:teuthology.orchestra.run.smithi191.stdout: ceph x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 6.6 k 2024-09-14T05:30:55.632 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-09-14T05:30:55.633 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mds x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 2.4 M 2024-09-14T05:30:55.633 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 1.5 M 2024-09-14T05:30:55.633 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4775.g7f24e589.el9 ceph-noarch 291 k 2024-09-14T05:30:55.633 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mon x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 4.8 M 2024-09-14T05:30:55.633 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-osd x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 17 M 2024-09-14T05:30:55.633 INFO:teuthology.orchestra.run.smithi191.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-14T05:30:55.633 INFO:teuthology.orchestra.run.smithi191.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-14T05:30:55.633 INFO:teuthology.orchestra.run.smithi191.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-14T05:30:55.633 INFO:teuthology.orchestra.run.smithi191.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-14T05:30:55.633 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-14T05:30:55.633 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-14T05:30:55.633 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-14T05:30:55.634 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-14T05:30:55.634 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-14T05:30:55.634 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-14T05:30:55.634 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-collections 2024-09-14T05:30:55.634 INFO:teuthology.orchestra.run.smithi191.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-14T05:30:55.634 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-14T05:30:55.634 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-14T05:30:55.634 INFO:teuthology.orchestra.run.smithi191.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-14T05:30:55.634 INFO:teuthology.orchestra.run.smithi191.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-14T05:30:55.634 INFO:teuthology.orchestra.run.smithi191.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-14T05:30:55.634 INFO:teuthology.orchestra.run.smithi191.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-14T05:30:55.634 INFO:teuthology.orchestra.run.smithi191.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-14T05:30:55.634 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-14T05:30:55.635 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-14T05:30:55.635 INFO:teuthology.orchestra.run.smithi191.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-14T05:30:55.635 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-14T05:30:55.635 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-14T05:30:55.635 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-14T05:30:55.635 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-14T05:30:55.635 INFO:teuthology.orchestra.run.smithi191.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-14T05:30:55.635 INFO:teuthology.orchestra.run.smithi191.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-14T05:30:55.635 INFO:teuthology.orchestra.run.smithi191.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-14T05:30:55.635 INFO:teuthology.orchestra.run.smithi191.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-14T05:30:55.635 INFO:teuthology.orchestra.run.smithi191.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-14T05:30:55.635 INFO:teuthology.orchestra.run.smithi191.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-14T05:30:55.635 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:30:55.635 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-09-14T05:30:55.636 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-14T05:30:55.636 INFO:teuthology.orchestra.run.smithi191.stdout:Install 37 Packages 2024-09-14T05:30:55.636 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:30:55.636 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 31 M 2024-09-14T05:30:55.636 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 109 M 2024-09-14T05:30:55.636 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-09-14T05:30:56.046 INFO:teuthology.orchestra.run.smithi191.stdout:(1/37): ceph-19.3.0-4775.g7f24e589.el9.x86_64.r 54 kB/s | 6.6 kB 00:00 2024-09-14T05:30:56.070 INFO:teuthology.orchestra.run.smithi143.stdout:(1/12): ceph-grafana-dashboards-19.3.0-4775.g7f 246 kB/s | 34 kB 00:00 2024-09-14T05:30:56.087 INFO:teuthology.orchestra.run.smithi143.stdout:(2/12): ceph-prometheus-alerts-19.3.0-4775.g7f2 105 kB/s | 16 kB 00:00 2024-09-14T05:30:56.254 INFO:teuthology.orchestra.run.smithi191.stdout:(2/37): ceph-mgr-19.3.0-4775.g7f24e589.el9.x86_ 4.6 MB/s | 1.5 MB 00:00 2024-09-14T05:30:56.287 INFO:teuthology.orchestra.run.smithi143.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.3 MB/s | 269 kB 00:00 2024-09-14T05:30:56.330 INFO:teuthology.orchestra.run.smithi191.stdout:(3/37): ceph-mds-19.3.0-4775.g7f24e589.el9.x86_ 5.9 MB/s | 2.4 MB 00:00 2024-09-14T05:30:56.346 INFO:teuthology.orchestra.run.smithi143.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.7 MB/s | 1.0 MB 00:00 2024-09-14T05:30:56.396 INFO:teuthology.orchestra.run.smithi143.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 11 MB/s | 548 kB 00:00 2024-09-14T05:30:56.413 INFO:teuthology.orchestra.run.smithi191.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4775.g7f24 3.4 MB/s | 291 kB 00:00 2024-09-14T05:30:56.422 INFO:teuthology.orchestra.run.smithi143.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 778 kB/s | 19 kB 00:00 2024-09-14T05:30:56.480 INFO:teuthology.orchestra.run.smithi143.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 35 MB/s | 2.0 MB 00:00 2024-09-14T05:30:56.506 INFO:teuthology.orchestra.run.smithi143.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 5.7 MB/s | 144 kB 00:00 2024-09-14T05:30:56.531 INFO:teuthology.orchestra.run.smithi143.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-14T05:30:56.565 INFO:teuthology.orchestra.run.smithi143.stdout:(10/12): protobuf-compiler-3.14.0-13.el9.x86_64 3.0 MB/s | 863 kB 00:00 2024-09-14T05:30:56.590 INFO:teuthology.orchestra.run.smithi143.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.1 MB/s | 188 kB 00:00 2024-09-14T05:30:56.747 INFO:teuthology.orchestra.run.smithi191.stdout:(5/37): ceph-mon-19.3.0-4775.g7f24e589.el9.x86_ 6.8 MB/s | 4.8 MB 00:00 2024-09-14T05:30:56.890 INFO:teuthology.orchestra.run.smithi143.stdout:(12/12): ceph-mgr-dashboard-19.3.0-4775.g7f24e5 15 MB/s | 14 MB 00:00 2024-09-14T05:30:56.894 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-09-14T05:30:56.894 INFO:teuthology.orchestra.run.smithi143.stdout:Total 13 MB/s | 19 MB 00:01 2024-09-14T05:30:56.945 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-09-14T05:30:56.964 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-09-14T05:30:57.095 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-09-14T05:30:57.096 INFO:teuthology.orchestra.run.smithi191.stdout:(6/37): ceph-osd-19.3.0-4775.g7f24e589.el9.x86_ 24 MB/s | 17 MB 00:00 2024-09-14T05:30:57.256 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-09-14T05:30:57.257 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-09-14T05:30:57.381 INFO:teuthology.orchestra.run.smithi191.stdout:(7/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 262 kB/s | 253 kB 00:00 2024-09-14T05:30:57.665 INFO:teuthology.orchestra.run.smithi191.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 156 kB/s | 106 kB 00:00 2024-09-14T05:30:57.851 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-09-14T05:30:57.932 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-14T05:30:57.976 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-14T05:30:58.007 INFO:teuthology.orchestra.run.smithi191.stdout:(9/37): python3-cryptography-36.0.1-4.el9.x86_6 998 kB/s | 1.2 MB 00:01 2024-09-14T05:30:58.022 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-14T05:30:58.053 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-14T05:30:58.073 INFO:teuthology.orchestra.run.smithi191.stdout:(10/37): python3-pycparser-2.20-6.el9.noarch.rp 195 kB/s | 135 kB 00:00 2024-09-14T05:30:58.152 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-14T05:30:58.199 INFO:teuthology.orchestra.run.smithi191.stdout:(11/37): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 188 kB 00:00 2024-09-14T05:30:58.241 INFO:teuthology.orchestra.run.smithi191.stdout:(12/37): python3-mako-1.1.4-6.el9.noarch.rpm 4.0 MB/s | 172 kB 00:00 2024-09-14T05:30:58.287 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-14T05:30:58.291 INFO:teuthology.orchestra.run.smithi191.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 819 kB/s | 35 kB 00:00 2024-09-14T05:30:58.333 INFO:teuthology.orchestra.run.smithi191.stdout:(14/37): python3-packaging-20.9-5.el9.noarch.rp 1.8 MB/s | 77 kB 00:00 2024-09-14T05:30:58.375 INFO:teuthology.orchestra.run.smithi191.stdout:(15/37): python3-pytz-2021.1-5.el9.noarch.rpm 1.2 MB/s | 51 kB 00:00 2024-09-14T05:30:58.381 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-14T05:30:58.400 INFO:teuthology.orchestra.run.smithi191.stdout:(16/37): python3-requests-2.25.1-8.el9.noarch.r 170 kB/s | 125 kB 00:00 2024-09-14T05:30:58.425 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-14T05:30:58.425 INFO:teuthology.orchestra.run.smithi191.stdout:(17/37): python3-toml-0.10.2-6.el9.noarch.rpm 829 kB/s | 42 kB 00:00 2024-09-14T05:30:58.453 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-14T05:30:58.467 INFO:teuthology.orchestra.run.smithi191.stdout:(18/37): luarocks-3.9.2-5.el9.noarch.rpm 3.6 MB/s | 151 kB 00:00 2024-09-14T05:30:58.501 INFO:teuthology.orchestra.run.smithi191.stdout:(19/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.3 MB/s | 43 kB 00:00 2024-09-14T05:30:58.506 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4775.g7f24e589.el9 10/12 2024-09-14T05:30:58.526 INFO:teuthology.orchestra.run.smithi191.stdout:(20/37): lua-devel-5.4.4-4.el9.x86_64.rpm 178 kB/s | 22 kB 00:00 2024-09-14T05:30:58.568 INFO:teuthology.orchestra.run.smithi191.stdout:(21/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.5 MB/s | 172 kB 00:00 2024-09-14T05:30:58.585 INFO:teuthology.orchestra.run.smithi191.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 724 kB/s | 11 kB 00:00 2024-09-14T05:30:58.602 INFO:teuthology.orchestra.run.smithi191.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 1.0 MB/s | 18 kB 00:00 2024-09-14T05:30:58.619 INFO:teuthology.orchestra.run.smithi191.stdout:(24/37): python3-jaraco-collections-3.0.0-8.el9 1.4 MB/s | 23 kB 00:00 2024-09-14T05:30:58.636 INFO:teuthology.orchestra.run.smithi191.stdout:(25/37): python3-jaraco-functools-3.5.0-2.el9.n 1.1 MB/s | 19 kB 00:00 2024-09-14T05:30:58.669 INFO:teuthology.orchestra.run.smithi191.stdout:(26/37): python3-cherrypy-18.6.1-2.el9.noarch.r 2.4 MB/s | 358 kB 00:00 2024-09-14T05:30:58.698 INFO:teuthology.orchestra.run.smithi191.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 317 kB/s | 20 kB 00:00 2024-09-14T05:30:58.723 INFO:teuthology.orchestra.run.smithi191.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 865 kB/s | 46 kB 00:00 2024-09-14T05:30:58.749 INFO:teuthology.orchestra.run.smithi191.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-14T05:30:58.774 INFO:teuthology.orchestra.run.smithi191.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 657 kB/s | 16 kB 00:00 2024-09-14T05:30:58.799 INFO:teuthology.orchestra.run.smithi191.stdout:(31/37): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-09-14T05:30:58.825 INFO:teuthology.orchestra.run.smithi191.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-09-14T05:30:58.850 INFO:teuthology.orchestra.run.smithi191.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 710 kB/s | 36 kB 00:00 2024-09-14T05:30:58.875 INFO:teuthology.orchestra.run.smithi191.stdout:(34/37): python3-urllib3-1.26.5-6.el9.noarch.rp 247 kB/s | 215 kB 00:00 2024-09-14T05:30:58.909 INFO:teuthology.orchestra.run.smithi191.stdout:(35/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-09-14T05:30:58.934 INFO:teuthology.orchestra.run.smithi191.stdout:(36/37): python3-zc-lockfile-2.0-10.el9.noarch. 342 kB/s | 20 kB 00:00 2024-09-14T05:30:58.959 INFO:teuthology.orchestra.run.smithi191.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-09-14T05:30:58.965 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-09-14T05:30:58.965 INFO:teuthology.orchestra.run.smithi191.stdout:Total 9.2 MB/s | 31 MB 00:03 2024-09-14T05:30:59.121 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-09-14T05:30:59.161 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-09-14T05:30:59.161 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-09-14T05:30:59.650 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-09-14T05:30:59.651 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-09-14T05:31:01.017 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-09-14T05:31:01.087 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-14T05:31:01.127 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-14T05:31:01.165 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-14T05:31:01.234 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-14T05:31:01.288 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-14T05:31:01.329 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-14T05:31:01.422 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-14T05:31:01.510 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-14T05:31:01.567 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-14T05:31:01.603 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-14T05:31:01.633 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-14T05:31:01.667 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-14T05:31:01.675 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4775.g7f24e589.el 11/12 2024-09-14T05:31:01.700 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-14T05:31:01.740 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4775.g7f24e589.el9.noa 12/12 2024-09-14T05:31:01.741 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-14T05:31:01.775 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-14T05:31:01.810 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-14T05:31:01.858 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-14T05:31:01.890 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-14T05:31:01.963 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-14T05:31:02.101 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-14T05:31:02.189 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-14T05:31:02.252 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-14T05:31:02.299 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-14T05:31:02.347 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-14T05:31:02.403 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-14T05:31:02.585 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-14T05:31:02.636 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-14T05:31:02.704 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-14T05:31:02.793 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-14T05:31:02.827 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-14T05:31:02.954 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-14T05:31:03.132 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4775.g7f24e589.el9. 32/37 2024-09-14T05:31:03.163 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-2:19.3.0-4775.g7f24e589.el9.x86_64 33/37 2024-09-14T05:31:03.188 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4775.g7f24e589.el9.x86_64 33/37 2024-09-14T05:31:03.188 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T05:31:03.188 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-14T05:31:03.188 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-14T05:31:03.188 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-14T05:31:03.188 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:31:03.602 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4775.g7f24e589.el9.noa 12/12 2024-09-14T05:31:03.602 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4775.g7f24e589.el 1/12 2024-09-14T05:31:03.602 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4775.g7f24e589.el9.noa 2/12 2024-09-14T05:31:03.602 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4775.g7f24e589.el9 3/12 2024-09-14T05:31:03.602 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-14T05:31:03.603 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-14T05:31:03.603 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-14T05:31:03.603 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-14T05:31:03.603 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-14T05:31:03.603 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-14T05:31:03.603 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-14T05:31:03.603 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-14T05:31:03.913 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-14T05:31:03.913 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:31:03.913 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-09-14T05:31:03.913 INFO:teuthology.orchestra.run.smithi143.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-14T05:31:03.913 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-grafana-dashboards-2:19.3.0-4775.g7f24e589.el9.noarch 2024-09-14T05:31:03.913 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-dashboard-2:19.3.0-4775.g7f24e589.el9.noarch 2024-09-14T05:31:03.913 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-prometheus-alerts-2:19.3.0-4775.g7f24e589.el9.noarch 2024-09-14T05:31:03.913 INFO:teuthology.orchestra.run.smithi143.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-14T05:31:03.913 INFO:teuthology.orchestra.run.smithi143.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-14T05:31:03.913 INFO:teuthology.orchestra.run.smithi143.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-14T05:31:03.913 INFO:teuthology.orchestra.run.smithi143.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-14T05:31:03.913 INFO:teuthology.orchestra.run.smithi143.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-14T05:31:03.914 INFO:teuthology.orchestra.run.smithi143.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-14T05:31:03.914 INFO:teuthology.orchestra.run.smithi143.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-14T05:31:03.914 INFO:teuthology.orchestra.run.smithi143.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-14T05:31:03.914 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:31:03.914 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-09-14T05:31:04.187 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-14T05:31:04.647 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-osd-2:19.3.0-4775.g7f24e589.el9.x86_64 34/37 2024-09-14T05:31:04.674 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-osd-2:19.3.0-4775.g7f24e589.el9.x86_64 34/37 2024-09-14T05:31:04.674 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T05:31:04.675 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-14T05:31:04.675 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-14T05:31:04.675 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-14T05:31:04.675 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:31:04.755 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:01:09 ago on Sat 14 Sep 2024 05:29:55 AM UTC. 2024-09-14T05:31:04.890 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-09-14T05:31:04.891 INFO:teuthology.orchestra.run.smithi143.stdout:======================================================================================= 2024-09-14T05:31:04.891 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repository Size 2024-09-14T05:31:04.891 INFO:teuthology.orchestra.run.smithi143.stdout:======================================================================================= 2024-09-14T05:31:04.891 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-09-14T05:31:04.891 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4775.g7f24e589.el9 ceph-noarch 7.4 M 2024-09-14T05:31:04.891 INFO:teuthology.orchestra.run.smithi143.stdout:Installing dependencies: 2024-09-14T05:31:04.891 INFO:teuthology.orchestra.run.smithi143.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-14T05:31:04.891 INFO:teuthology.orchestra.run.smithi143.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-14T05:31:04.891 INFO:teuthology.orchestra.run.smithi143.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-14T05:31:04.891 INFO:teuthology.orchestra.run.smithi143.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-14T05:31:04.891 INFO:teuthology.orchestra.run.smithi143.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-14T05:31:04.891 INFO:teuthology.orchestra.run.smithi143.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-14T05:31:04.892 INFO:teuthology.orchestra.run.smithi143.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-14T05:31:04.892 INFO:teuthology.orchestra.run.smithi143.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-14T05:31:04.892 INFO:teuthology.orchestra.run.smithi143.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-14T05:31:04.892 INFO:teuthology.orchestra.run.smithi143.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-14T05:31:04.892 INFO:teuthology.orchestra.run.smithi143.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-14T05:31:04.892 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:31:04.892 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-09-14T05:31:04.892 INFO:teuthology.orchestra.run.smithi143.stdout:======================================================================================= 2024-09-14T05:31:04.892 INFO:teuthology.orchestra.run.smithi143.stdout:Install 12 Packages 2024-09-14T05:31:04.892 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:31:04.893 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 38 M 2024-09-14T05:31:04.893 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 212 M 2024-09-14T05:31:04.893 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-09-14T05:31:05.131 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mon-2:19.3.0-4775.g7f24e589.el9.x86_64 35/37 2024-09-14T05:31:05.155 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mon-2:19.3.0-4775.g7f24e589.el9.x86_64 35/37 2024-09-14T05:31:05.155 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T05:31:05.155 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-14T05:31:05.155 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-14T05:31:05.155 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-14T05:31:05.155 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:31:05.382 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mds-2:19.3.0-4775.g7f24e589.el9.x86_64 36/37 2024-09-14T05:31:05.405 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mds-2:19.3.0-4775.g7f24e589.el9.x86_64 36/37 2024-09-14T05:31:05.405 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T05:31:05.405 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-14T05:31:05.405 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-14T05:31:05.405 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-14T05:31:05.406 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:31:05.414 INFO:teuthology.orchestra.run.smithi143.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.3 MB/s | 191 kB 00:00 2024-09-14T05:31:05.479 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-2:19.3.0-4775.g7f24e589.el9.x86_64 37/37 2024-09-14T05:31:05.497 INFO:teuthology.orchestra.run.smithi143.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.5 MB/s | 801 kB 00:00 2024-09-14T05:31:05.523 INFO:teuthology.orchestra.run.smithi143.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 301 kB/s | 33 kB 00:00 2024-09-14T05:31:05.565 INFO:teuthology.orchestra.run.smithi143.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 436 kB/s | 18 kB 00:00 2024-09-14T05:31:05.599 INFO:teuthology.orchestra.run.smithi143.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 1.1 MB/s | 38 kB 00:00 2024-09-14T05:31:05.749 INFO:teuthology.orchestra.run.smithi143.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-09-14T05:31:05.849 INFO:teuthology.orchestra.run.smithi143.stdout:(7/12): ceph-mgr-diskprediction-local-19.3.0-47 13 MB/s | 7.4 MB 00:00 2024-09-14T05:31:05.874 INFO:teuthology.orchestra.run.smithi143.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.9 MB/s | 245 kB 00:00 2024-09-14T05:31:05.916 INFO:teuthology.orchestra.run.smithi143.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.9 MB/s | 250 kB 00:00 2024-09-14T05:31:06.025 INFO:teuthology.orchestra.run.smithi143.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 12 MB/s | 4.9 MB 00:00 2024-09-14T05:31:06.309 INFO:teuthology.orchestra.run.smithi143.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 41 MB/s | 16 MB 00:00 2024-09-14T05:31:06.567 INFO:teuthology.orchestra.run.smithi143.stdout:(12/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 7.1 MB/s | 5.1 MB 00:00 2024-09-14T05:31:06.568 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-09-14T05:31:06.568 INFO:teuthology.orchestra.run.smithi143.stdout:Total 23 MB/s | 38 MB 00:01 2024-09-14T05:31:06.923 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-09-14T05:31:06.949 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-09-14T05:31:06.949 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-09-14T05:31:07.305 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-2:19.3.0-4775.g7f24e589.el9.x86_64 37/37 2024-09-14T05:31:07.305 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-2:19.3.0-4775.g7f24e589.el9.x86_64 1/37 2024-09-14T05:31:07.305 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mds-2:19.3.0-4775.g7f24e589.el9.x86_64 2/37 2024-09-14T05:31:07.305 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-2:19.3.0-4775.g7f24e589.el9.x86_64 3/37 2024-09-14T05:31:07.305 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mon-2:19.3.0-4775.g7f24e589.el9.x86_64 4/37 2024-09-14T05:31:07.305 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-osd-2:19.3.0-4775.g7f24e589.el9.x86_64 5/37 2024-09-14T05:31:07.306 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4775.g7f24e589.el9. 6/37 2024-09-14T05:31:07.306 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-14T05:31:07.306 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-14T05:31:07.306 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-14T05:31:07.306 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-14T05:31:07.306 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-14T05:31:07.306 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-14T05:31:07.306 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-14T05:31:07.306 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-14T05:31:07.306 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-14T05:31:07.306 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-14T05:31:07.306 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-14T05:31:07.307 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-14T05:31:07.307 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-14T05:31:07.307 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-14T05:31:07.307 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-14T05:31:07.307 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-14T05:31:07.307 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-14T05:31:07.307 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-14T05:31:07.307 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-14T05:31:07.307 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-14T05:31:07.307 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-14T05:31:07.307 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-14T05:31:07.307 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-14T05:31:07.307 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-14T05:31:07.308 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-14T05:31:07.308 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-14T05:31:07.308 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-14T05:31:07.309 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-14T05:31:07.309 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-14T05:31:07.309 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-14T05:31:07.374 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-09-14T05:31:07.375 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-09-14T05:31:08.091 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-14T05:31:08.092 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:31:08.092 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-09-14T05:31:08.092 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T05:31:08.092 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mds-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T05:31:08.092 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T05:31:08.092 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-modules-core-2:19.3.0-4775.g7f24e589.el9.noarch 2024-09-14T05:31:08.092 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mon-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T05:31:08.092 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-osd-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T05:31:08.092 INFO:teuthology.orchestra.run.smithi191.stdout: lua-5.4.4-4.el9.x86_64 2024-09-14T05:31:08.092 INFO:teuthology.orchestra.run.smithi191.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-14T05:31:08.092 INFO:teuthology.orchestra.run.smithi191.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-14T05:31:08.092 INFO:teuthology.orchestra.run.smithi191.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-14T05:31:08.092 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-14T05:31:08.093 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-14T05:31:08.093 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-14T05:31:08.093 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-14T05:31:08.093 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-14T05:31:08.093 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-14T05:31:08.093 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-14T05:31:08.093 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-14T05:31:08.093 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-14T05:31:08.093 INFO:teuthology.orchestra.run.smithi191.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-14T05:31:08.093 INFO:teuthology.orchestra.run.smithi191.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-14T05:31:08.093 INFO:teuthology.orchestra.run.smithi191.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-14T05:31:08.093 INFO:teuthology.orchestra.run.smithi191.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-14T05:31:08.093 INFO:teuthology.orchestra.run.smithi191.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-14T05:31:08.094 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-14T05:31:08.094 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ply-3.11-14.el9.noarch 2024-09-14T05:31:08.094 INFO:teuthology.orchestra.run.smithi191.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-14T05:31:08.094 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-14T05:31:08.094 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-14T05:31:08.094 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-14T05:31:08.094 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-14T05:31:08.094 INFO:teuthology.orchestra.run.smithi191.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-14T05:31:08.094 INFO:teuthology.orchestra.run.smithi191.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-14T05:31:08.094 INFO:teuthology.orchestra.run.smithi191.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-14T05:31:08.094 INFO:teuthology.orchestra.run.smithi191.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-14T05:31:08.094 INFO:teuthology.orchestra.run.smithi191.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-14T05:31:08.094 INFO:teuthology.orchestra.run.smithi191.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-14T05:31:08.094 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:31:08.095 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-14T05:31:08.140 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-09-14T05:31:08.216 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-14T05:31:08.257 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-14T05:31:08.354 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-14T05:31:08.391 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-14T05:31:08.468 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-base 2024-09-14T05:31:08.577 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-14T05:31:08.607 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-14T05:31:08.683 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-14T05:31:09.037 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:55 ago on Sat 14 Sep 2024 05:30:14 AM UTC. 2024-09-14T05:31:09.123 INFO:teuthology.orchestra.run.smithi191.stdout:Package ceph-base-2:19.3.0-4775.g7f24e589.el9.x86_64 is already installed. 2024-09-14T05:31:09.165 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-14T05:31:09.166 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-09-14T05:31:09.166 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-14T05:31:09.243 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install cephadm 2024-09-14T05:31:09.347 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-14T05:31:09.462 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-14T05:31:09.812 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:55 ago on Sat 14 Sep 2024 05:30:14 AM UTC. 2024-09-14T05:31:09.949 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-14T05:31:09.949 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-14T05:31:09.949 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-09-14T05:31:09.950 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-14T05:31:09.950 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-09-14T05:31:09.950 INFO:teuthology.orchestra.run.smithi191.stdout: cephadm noarch 2:19.3.0-4775.g7f24e589.el9 ceph-noarch 783 k 2024-09-14T05:31:09.950 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:31:09.950 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-09-14T05:31:09.950 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-14T05:31:09.951 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-09-14T05:31:09.951 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:31:09.951 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 783 k 2024-09-14T05:31:09.951 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 789 k 2024-09-14T05:31:09.951 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-09-14T05:31:10.223 INFO:teuthology.orchestra.run.smithi191.stdout:cephadm-19.3.0-4775.g7f24e589.el9.noarch.rpm 2.8 MB/s | 783 kB 00:00 2024-09-14T05:31:10.224 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-09-14T05:31:10.224 INFO:teuthology.orchestra.run.smithi191.stdout:Total 2.8 MB/s | 783 kB 00:00 2024-09-14T05:31:10.224 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-09-14T05:31:10.229 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-09-14T05:31:10.229 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-09-14T05:31:10.235 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-09-14T05:31:10.235 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-09-14T05:31:10.326 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-09-14T05:31:10.690 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-14T05:31:10.789 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: cephadm-2:19.3.0-4775.g7f24e589.el9.noarch 1/1 2024-09-14T05:31:10.868 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : cephadm-2:19.3.0-4775.g7f24e589.el9.noarch 1/1 2024-09-14T05:31:11.238 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: cephadm-2:19.3.0-4775.g7f24e589.el9.noarch 1/1 2024-09-14T05:31:11.478 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : cephadm-2:19.3.0-4775.g7f24e589.el9.noarch 1/1 2024-09-14T05:31:11.479 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:31:11.479 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-09-14T05:31:11.479 INFO:teuthology.orchestra.run.smithi191.stdout: cephadm-2:19.3.0-4775.g7f24e589.el9.noarch 2024-09-14T05:31:11.479 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:31:11.479 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-14T05:31:11.705 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-immutable-object-cache 2024-09-14T05:31:11.736 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-14T05:31:11.766 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4775.g7f24e 12/12 2024-09-14T05:31:12.285 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:58 ago on Sat 14 Sep 2024 05:30:14 AM UTC. 2024-09-14T05:31:12.423 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-14T05:31:12.423 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-14T05:31:12.424 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repo Size 2024-09-14T05:31:12.424 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-14T05:31:12.424 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-09-14T05:31:12.424 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 145 k 2024-09-14T05:31:12.424 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:31:12.424 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-09-14T05:31:12.424 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-14T05:31:12.424 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-09-14T05:31:12.424 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:31:12.425 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 145 k 2024-09-14T05:31:12.425 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 432 k 2024-09-14T05:31:12.425 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-09-14T05:31:12.638 INFO:teuthology.orchestra.run.smithi191.stdout:ceph-immutable-object-cache-19.3.0-4775.g7f24e5 683 kB/s | 145 kB 00:00 2024-09-14T05:31:12.638 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-09-14T05:31:12.639 INFO:teuthology.orchestra.run.smithi191.stdout:Total 677 kB/s | 145 kB 00:00 2024-09-14T05:31:12.639 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-09-14T05:31:12.646 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-09-14T05:31:12.646 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-09-14T05:31:12.703 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-09-14T05:31:12.703 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-09-14T05:31:12.848 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-09-14T05:31:12.905 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4775.g7f24e589. 1/1 2024-09-14T05:31:12.930 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4775.g7f24e589. 1/1 2024-09-14T05:31:12.930 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T05:31:12.931 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-14T05:31:12.931 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:31:13.927 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4775.g7f24e589. 1/1 2024-09-14T05:31:13.928 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:31:13.928 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-09-14T05:31:13.928 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-immutable-object-cache-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T05:31:13.928 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:31:13.928 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-14T05:31:14.140 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-mgr 2024-09-14T05:31:14.377 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4775.g7f24e 12/12 2024-09-14T05:31:14.377 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4775.g7f24e 1/12 2024-09-14T05:31:14.377 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-14T05:31:14.377 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-14T05:31:14.377 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-14T05:31:14.378 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-14T05:31:14.378 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-14T05:31:14.378 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-14T05:31:14.378 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-14T05:31:14.378 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-14T05:31:14.378 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-14T05:31:14.378 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-14T05:31:14.737 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:00 ago on Sat 14 Sep 2024 05:30:14 AM UTC. 2024-09-14T05:31:14.829 INFO:teuthology.orchestra.run.smithi191.stdout:Package ceph-mgr-2:19.3.0-4775.g7f24e589.el9.x86_64 is already installed. 2024-09-14T05:31:14.833 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-14T05:31:14.833 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:31:14.833 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-09-14T05:31:14.833 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4775.g7f24e589.el9.noarch 2024-09-14T05:31:14.833 INFO:teuthology.orchestra.run.smithi143.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-14T05:31:14.833 INFO:teuthology.orchestra.run.smithi143.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-14T05:31:14.833 INFO:teuthology.orchestra.run.smithi143.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-14T05:31:14.833 INFO:teuthology.orchestra.run.smithi143.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-14T05:31:14.833 INFO:teuthology.orchestra.run.smithi143.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-14T05:31:14.833 INFO:teuthology.orchestra.run.smithi143.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-14T05:31:14.833 INFO:teuthology.orchestra.run.smithi143.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-14T05:31:14.834 INFO:teuthology.orchestra.run.smithi143.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-14T05:31:14.834 INFO:teuthology.orchestra.run.smithi143.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-14T05:31:14.834 INFO:teuthology.orchestra.run.smithi143.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-14T05:31:14.834 INFO:teuthology.orchestra.run.smithi143.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-14T05:31:14.834 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:31:14.834 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-09-14T05:31:14.872 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-14T05:31:14.873 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-09-14T05:31:14.873 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-14T05:31:14.942 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-mgr-dashboard 2024-09-14T05:31:15.213 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph-mgr-rook 2024-09-14T05:31:15.515 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:01 ago on Sat 14 Sep 2024 05:30:14 AM UTC. 2024-09-14T05:31:15.649 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-14T05:31:15.650 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-14T05:31:15.650 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-09-14T05:31:15.650 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-14T05:31:15.650 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-09-14T05:31:15.650 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4775.g7f24e589.el9 ceph-noarch 14 M 2024-09-14T05:31:15.650 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-09-14T05:31:15.650 INFO:teuthology.orchestra.run.smithi191.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-14T05:31:15.650 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4775.g7f24e589.el9 ceph-noarch 34 k 2024-09-14T05:31:15.651 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4775.g7f24e589.el9 ceph-noarch 16 k 2024-09-14T05:31:15.651 INFO:teuthology.orchestra.run.smithi191.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-14T05:31:15.651 INFO:teuthology.orchestra.run.smithi191.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-14T05:31:15.651 INFO:teuthology.orchestra.run.smithi191.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-14T05:31:15.651 INFO:teuthology.orchestra.run.smithi191.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-14T05:31:15.651 INFO:teuthology.orchestra.run.smithi191.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-14T05:31:15.651 INFO:teuthology.orchestra.run.smithi191.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-14T05:31:15.651 INFO:teuthology.orchestra.run.smithi191.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-14T05:31:15.651 INFO:teuthology.orchestra.run.smithi191.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-14T05:31:15.651 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:31:15.651 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-09-14T05:31:15.651 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-14T05:31:15.651 INFO:teuthology.orchestra.run.smithi191.stdout:Install 12 Packages 2024-09-14T05:31:15.651 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:31:15.652 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 19 M 2024-09-14T05:31:15.652 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 159 M 2024-09-14T05:31:15.652 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-09-14T05:31:15.785 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:01:20 ago on Sat 14 Sep 2024 05:29:55 AM UTC. 2024-09-14T05:31:15.920 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-09-14T05:31:15.921 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-09-14T05:31:15.921 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repository Size 2024-09-14T05:31:15.921 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-09-14T05:31:15.921 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-09-14T05:31:15.921 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-rook noarch 2:19.3.0-4775.g7f24e589.el9 ceph-noarch 49 k 2024-09-14T05:31:15.921 INFO:teuthology.orchestra.run.smithi143.stdout:Installing dependencies: 2024-09-14T05:31:15.921 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-14T05:31:15.921 INFO:teuthology.orchestra.run.smithi143.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-14T05:31:15.921 INFO:teuthology.orchestra.run.smithi143.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-14T05:31:15.921 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-14T05:31:15.921 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-14T05:31:15.922 INFO:teuthology.orchestra.run.smithi143.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-14T05:31:15.922 INFO:teuthology.orchestra.run.smithi143.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-14T05:31:15.922 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-14T05:31:15.922 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-14T05:31:15.922 INFO:teuthology.orchestra.run.smithi143.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-14T05:31:15.922 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-14T05:31:15.922 INFO:teuthology.orchestra.run.smithi143.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-14T05:31:15.922 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:31:15.922 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-09-14T05:31:15.922 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-09-14T05:31:15.922 INFO:teuthology.orchestra.run.smithi143.stdout:Install 13 Packages 2024-09-14T05:31:15.922 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:31:15.923 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 2.2 M 2024-09-14T05:31:15.923 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 26 M 2024-09-14T05:31:15.923 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-09-14T05:31:16.227 INFO:teuthology.orchestra.run.smithi143.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 233 kB/s | 26 kB 00:00 2024-09-14T05:31:16.252 INFO:teuthology.orchestra.run.smithi143.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 142 kB/s | 19 kB 00:00 2024-09-14T05:31:16.277 INFO:teuthology.orchestra.run.smithi143.stdout:(3/13): ceph-mgr-rook-19.3.0-4775.g7f24e589.el9 307 kB/s | 49 kB 00:00 2024-09-14T05:31:16.336 INFO:teuthology.orchestra.run.smithi143.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 222 kB 00:00 2024-09-14T05:31:16.361 INFO:teuthology.orchestra.run.smithi143.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-09-14T05:31:16.395 INFO:teuthology.orchestra.run.smithi143.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 914 kB/s | 54 kB 00:00 2024-09-14T05:31:16.428 INFO:teuthology.orchestra.run.smithi143.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 481 kB/s | 32 kB 00:00 2024-09-14T05:31:16.454 INFO:teuthology.orchestra.run.smithi143.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 247 kB/s | 14 kB 00:00 2024-09-14T05:31:16.479 INFO:teuthology.orchestra.run.smithi143.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.4 MB/s | 279 kB 00:00 2024-09-14T05:31:16.513 INFO:teuthology.orchestra.run.smithi143.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.6 MB/s | 220 kB 00:00 2024-09-14T05:31:16.533 INFO:teuthology.orchestra.run.smithi191.stdout:(1/12): ceph-prometheus-alerts-19.3.0-4775.g7f2 104 kB/s | 16 kB 00:00 2024-09-14T05:31:16.538 INFO:teuthology.orchestra.run.smithi143.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.8 MB/s | 90 kB 00:00 2024-09-14T05:31:16.558 INFO:teuthology.orchestra.run.smithi191.stdout:(2/12): ceph-grafana-dashboards-19.3.0-4775.g7f 186 kB/s | 34 kB 00:00 2024-09-14T05:31:16.563 INFO:teuthology.orchestra.run.smithi143.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 706 kB/s | 59 kB 00:00 2024-09-14T05:31:16.605 INFO:teuthology.orchestra.run.smithi143.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.8 MB/s | 1.0 MB 00:00 2024-09-14T05:31:16.607 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-09-14T05:31:16.607 INFO:teuthology.orchestra.run.smithi143.stdout:Total 3.2 MB/s | 2.2 MB 00:00 2024-09-14T05:31:16.671 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-09-14T05:31:16.687 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-09-14T05:31:16.687 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-09-14T05:31:16.734 INFO:teuthology.orchestra.run.smithi191.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.5 MB/s | 269 kB 00:00 2024-09-14T05:31:16.776 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-09-14T05:31:16.776 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-09-14T05:31:16.784 INFO:teuthology.orchestra.run.smithi191.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.0 MB/s | 1.0 MB 00:00 2024-09-14T05:31:16.842 INFO:teuthology.orchestra.run.smithi191.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 9.2 MB/s | 548 kB 00:00 2024-09-14T05:31:16.868 INFO:teuthology.orchestra.run.smithi191.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 783 kB/s | 19 kB 00:00 2024-09-14T05:31:17.035 INFO:teuthology.orchestra.run.smithi191.stdout:(7/12): ceph-mgr-dashboard-19.3.0-4775.g7f24e58 21 MB/s | 14 MB 00:00 2024-09-14T05:31:17.110 INFO:teuthology.orchestra.run.smithi191.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 8.4 MB/s | 2.0 MB 00:00 2024-09-14T05:31:17.127 INFO:teuthology.orchestra.run.smithi191.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 2.1 MB/s | 31 kB 00:00 2024-09-14T05:31:17.152 INFO:teuthology.orchestra.run.smithi191.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 7.3 MB/s | 188 kB 00:00 2024-09-14T05:31:17.186 INFO:teuthology.orchestra.run.smithi191.stdout:(11/12): python3-grpcio-tools-1.46.7-10.el9.x86 959 kB/s | 144 kB 00:00 2024-09-14T05:31:17.219 INFO:teuthology.orchestra.run.smithi191.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 1.7 MB/s | 863 kB 00:00 2024-09-14T05:31:17.223 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-09-14T05:31:17.223 INFO:teuthology.orchestra.run.smithi191.stdout:Total 12 MB/s | 19 MB 00:01 2024-09-14T05:31:17.270 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-09-14T05:31:17.279 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-09-14T05:31:17.289 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-09-14T05:31:17.289 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-09-14T05:31:17.453 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-14T05:31:17.514 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-14T05:31:17.557 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-09-14T05:31:17.557 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-09-14T05:31:17.570 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-14T05:31:17.607 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-14T05:31:17.649 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-14T05:31:17.759 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-14T05:31:17.860 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-14T05:31:17.909 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-14T05:31:18.144 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-09-14T05:31:18.211 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-14T05:31:18.255 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-14T05:31:18.302 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-14T05:31:18.332 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-14T05:31:18.439 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-14T05:31:18.574 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-14T05:31:18.624 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-14T05:31:18.668 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-14T05:31:18.673 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-14T05:31:18.707 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-14T05:31:18.728 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-14T05:31:18.755 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-14T05:31:18.757 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-14T05:31:18.777 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mgr-rook-2:19.3.0-4775.g7f24e589.el9.noarch 13/13 2024-09-14T05:31:18.789 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4775.g7f24e589.el9 10/12 2024-09-14T05:31:19.524 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4775.g7f24e589.el9.noarch 13/13 2024-09-14T05:31:19.524 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4775.g7f24e589.el9.noarch 1/13 2024-09-14T05:31:19.524 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-14T05:31:19.524 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-14T05:31:19.524 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-14T05:31:19.524 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-14T05:31:19.524 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-14T05:31:19.524 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-14T05:31:19.524 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-14T05:31:19.524 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-14T05:31:19.524 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-14T05:31:19.525 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-14T05:31:19.525 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-14T05:31:19.870 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-14T05:31:19.870 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:31:19.870 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-09-14T05:31:19.870 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-rook-2:19.3.0-4775.g7f24e589.el9.noarch 2024-09-14T05:31:19.870 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-14T05:31:19.870 INFO:teuthology.orchestra.run.smithi143.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-14T05:31:19.871 INFO:teuthology.orchestra.run.smithi143.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-14T05:31:19.871 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-14T05:31:19.871 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-14T05:31:19.871 INFO:teuthology.orchestra.run.smithi143.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-14T05:31:19.871 INFO:teuthology.orchestra.run.smithi143.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-14T05:31:19.871 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-14T05:31:19.871 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-14T05:31:19.871 INFO:teuthology.orchestra.run.smithi143.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-14T05:31:19.871 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-14T05:31:19.871 INFO:teuthology.orchestra.run.smithi143.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-14T05:31:19.871 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:31:19.871 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-09-14T05:31:20.123 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph-mgr-cephadm 2024-09-14T05:31:20.693 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:01:25 ago on Sat 14 Sep 2024 05:29:55 AM UTC. 2024-09-14T05:31:20.828 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-09-14T05:31:20.829 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-09-14T05:31:20.830 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repository Size 2024-09-14T05:31:20.830 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-09-14T05:31:20.830 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-09-14T05:31:20.830 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4775.g7f24e589.el9 ceph-noarch 154 k 2024-09-14T05:31:20.830 INFO:teuthology.orchestra.run.smithi143.stdout:Installing dependencies: 2024-09-14T05:31:20.830 INFO:teuthology.orchestra.run.smithi143.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-14T05:31:20.830 INFO:teuthology.orchestra.run.smithi143.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-14T05:31:20.830 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-14T05:31:20.830 INFO:teuthology.orchestra.run.smithi143.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-14T05:31:20.830 INFO:teuthology.orchestra.run.smithi143.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-14T05:31:20.830 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:31:20.830 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-09-14T05:31:20.831 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-09-14T05:31:20.831 INFO:teuthology.orchestra.run.smithi143.stdout:Install 6 Packages 2024-09-14T05:31:20.831 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:31:20.831 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 7.0 M 2024-09-14T05:31:20.831 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 33 M 2024-09-14T05:31:20.831 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-09-14T05:31:21.224 INFO:teuthology.orchestra.run.smithi143.stdout:(1/6): ceph-mgr-cephadm-19.3.0-4775.g7f24e589.e 768 kB/s | 154 kB 00:00 2024-09-14T05:31:21.274 INFO:teuthology.orchestra.run.smithi143.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 11 MB/s | 548 kB 00:00 2024-09-14T05:31:21.291 INFO:teuthology.orchestra.run.smithi143.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 3.4 MB/s | 58 kB 00:00 2024-09-14T05:31:21.308 INFO:teuthology.orchestra.run.smithi143.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 4.3 MB/s | 74 kB 00:00 2024-09-14T05:31:21.659 INFO:teuthology.orchestra.run.smithi143.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 391 kB/s | 248 kB 00:00 2024-09-14T05:31:21.959 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4775.g7f24e589.el 11/12 2024-09-14T05:31:22.010 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4775.g7f24e589.el9.noa 12/12 2024-09-14T05:31:24.001 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4775.g7f24e589.el9.noa 12/12 2024-09-14T05:31:24.001 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4775.g7f24e589.el 1/12 2024-09-14T05:31:24.001 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4775.g7f24e589.el9.noa 2/12 2024-09-14T05:31:24.002 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4775.g7f24e589.el9 3/12 2024-09-14T05:31:24.002 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-14T05:31:24.002 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-14T05:31:24.002 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-14T05:31:24.002 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-14T05:31:24.002 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-14T05:31:24.002 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-14T05:31:24.002 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-14T05:31:24.002 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-14T05:31:24.283 INFO:teuthology.orchestra.run.smithi143.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 1.8 MB/s | 6.0 MB 00:03 2024-09-14T05:31:24.286 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-09-14T05:31:24.286 INFO:teuthology.orchestra.run.smithi143.stdout:Total 2.0 MB/s | 7.0 MB 00:03 2024-09-14T05:31:24.335 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-14T05:31:24.335 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:31:24.335 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-09-14T05:31:24.335 INFO:teuthology.orchestra.run.smithi191.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-14T05:31:24.335 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-grafana-dashboards-2:19.3.0-4775.g7f24e589.el9.noarch 2024-09-14T05:31:24.335 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-dashboard-2:19.3.0-4775.g7f24e589.el9.noarch 2024-09-14T05:31:24.335 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-prometheus-alerts-2:19.3.0-4775.g7f24e589.el9.noarch 2024-09-14T05:31:24.335 INFO:teuthology.orchestra.run.smithi191.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-14T05:31:24.336 INFO:teuthology.orchestra.run.smithi191.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-14T05:31:24.336 INFO:teuthology.orchestra.run.smithi191.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-14T05:31:24.336 INFO:teuthology.orchestra.run.smithi191.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-14T05:31:24.336 INFO:teuthology.orchestra.run.smithi191.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-14T05:31:24.336 INFO:teuthology.orchestra.run.smithi191.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-14T05:31:24.336 INFO:teuthology.orchestra.run.smithi191.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-14T05:31:24.336 INFO:teuthology.orchestra.run.smithi191.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-14T05:31:24.336 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:31:24.336 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-14T05:31:24.359 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-09-14T05:31:24.371 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-09-14T05:31:24.371 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-09-14T05:31:24.471 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-09-14T05:31:24.471 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-09-14T05:31:24.592 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-14T05:31:24.775 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-09-14T05:31:24.930 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-14T05:31:24.977 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-14T05:31:25.155 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:11 ago on Sat 14 Sep 2024 05:30:14 AM UTC. 2024-09-14T05:31:25.288 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-14T05:31:25.289 INFO:teuthology.orchestra.run.smithi191.stdout:======================================================================================= 2024-09-14T05:31:25.290 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-09-14T05:31:25.290 INFO:teuthology.orchestra.run.smithi191.stdout:======================================================================================= 2024-09-14T05:31:25.290 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-09-14T05:31:25.290 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4775.g7f24e589.el9 ceph-noarch 7.4 M 2024-09-14T05:31:25.290 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-09-14T05:31:25.290 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-14T05:31:25.290 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-14T05:31:25.290 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-14T05:31:25.290 INFO:teuthology.orchestra.run.smithi191.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-14T05:31:25.290 INFO:teuthology.orchestra.run.smithi191.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-14T05:31:25.290 INFO:teuthology.orchestra.run.smithi191.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-14T05:31:25.290 INFO:teuthology.orchestra.run.smithi191.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-14T05:31:25.291 INFO:teuthology.orchestra.run.smithi191.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-14T05:31:25.291 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-14T05:31:25.291 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-14T05:31:25.291 INFO:teuthology.orchestra.run.smithi191.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-14T05:31:25.291 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:31:25.291 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-09-14T05:31:25.291 INFO:teuthology.orchestra.run.smithi191.stdout:======================================================================================= 2024-09-14T05:31:25.291 INFO:teuthology.orchestra.run.smithi191.stdout:Install 12 Packages 2024-09-14T05:31:25.291 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:31:25.292 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 38 M 2024-09-14T05:31:25.292 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 212 M 2024-09-14T05:31:25.292 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-09-14T05:31:25.496 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-14T05:31:25.588 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-14T05:31:25.664 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-14T05:31:25.689 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4775.g7f24e589.el9.noarch 6/6 2024-09-14T05:31:26.276 INFO:teuthology.orchestra.run.smithi191.stdout:(1/12): ceph-mgr-diskprediction-local-19.3.0-47 16 MB/s | 7.4 MB 00:00 2024-09-14T05:31:26.384 INFO:teuthology.orchestra.run.smithi191.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 303 kB/s | 33 kB 00:00 2024-09-14T05:31:26.476 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4775.g7f24e589.el9.noarch 6/6 2024-09-14T05:31:26.476 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4775.g7f24e589.el9.noarch 1/6 2024-09-14T05:31:26.476 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-14T05:31:26.476 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-14T05:31:26.476 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-14T05:31:26.476 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-14T05:31:26.636 INFO:teuthology.orchestra.run.smithi191.stdout:(3/12): libquadmath-11.5.0-2.el9.x86_64.rpm 235 kB/s | 191 kB 00:00 2024-09-14T05:31:26.710 INFO:teuthology.orchestra.run.smithi191.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 245 kB/s | 18 kB 00:00 2024-09-14T05:31:26.742 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-14T05:31:26.742 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:31:26.742 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-09-14T05:31:26.742 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-cephadm-2:19.3.0-4775.g7f24e589.el9.noarch 2024-09-14T05:31:26.742 INFO:teuthology.orchestra.run.smithi143.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-14T05:31:26.742 INFO:teuthology.orchestra.run.smithi143.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-14T05:31:26.742 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-14T05:31:26.742 INFO:teuthology.orchestra.run.smithi143.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-14T05:31:26.742 INFO:teuthology.orchestra.run.smithi143.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-14T05:31:26.742 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:31:26.742 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-09-14T05:31:26.793 INFO:teuthology.orchestra.run.smithi191.stdout:(5/12): libgfortran-11.5.0-2.el9.x86_64.rpm 823 kB/s | 801 kB 00:00 2024-09-14T05:31:26.810 INFO:teuthology.orchestra.run.smithi191.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 384 kB/s | 38 kB 00:00 2024-09-14T05:31:27.009 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph-fuse 2024-09-14T05:31:27.011 INFO:teuthology.orchestra.run.smithi191.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.2 MB/s | 245 kB 00:00 2024-09-14T05:31:27.128 INFO:teuthology.orchestra.run.smithi191.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.0 MB/s | 3.0 MB 00:00 2024-09-14T05:31:27.220 INFO:teuthology.orchestra.run.smithi191.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.7 MB/s | 250 kB 00:00 2024-09-14T05:31:27.570 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:01:32 ago on Sat 14 Sep 2024 05:29:55 AM UTC. 2024-09-14T05:31:27.704 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-09-14T05:31:27.705 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-09-14T05:31:27.705 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repository Size 2024-09-14T05:31:27.705 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-09-14T05:31:27.705 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-09-14T05:31:27.705 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-fuse x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 900 k 2024-09-14T05:31:27.705 INFO:teuthology.orchestra.run.smithi143.stdout:Installing dependencies: 2024-09-14T05:31:27.706 INFO:teuthology.orchestra.run.smithi143.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-14T05:31:27.706 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:31:27.706 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-09-14T05:31:27.706 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-09-14T05:31:27.706 INFO:teuthology.orchestra.run.smithi143.stdout:Install 2 Packages 2024-09-14T05:31:27.706 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:31:27.706 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 979 k 2024-09-14T05:31:27.706 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 2.9 M 2024-09-14T05:31:27.706 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-09-14T05:31:27.911 INFO:teuthology.orchestra.run.smithi143.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 502 kB/s | 80 kB 00:00 2024-09-14T05:31:28.053 INFO:teuthology.orchestra.run.smithi143.stdout:(2/2): ceph-fuse-19.3.0-4775.g7f24e589.el9.x86_ 2.9 MB/s | 900 kB 00:00 2024-09-14T05:31:28.053 INFO:teuthology.orchestra.run.smithi191.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 3.9 MB/s | 4.9 MB 00:01 2024-09-14T05:31:28.054 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-09-14T05:31:28.054 INFO:teuthology.orchestra.run.smithi143.stdout:Total 2.7 MB/s | 979 kB 00:00 2024-09-14T05:31:28.065 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-09-14T05:31:28.087 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-09-14T05:31:28.087 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-09-14T05:31:28.153 INFO:teuthology.orchestra.run.smithi191.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.5 MB/s | 5.1 MB 00:01 2024-09-14T05:31:28.167 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-09-14T05:31:28.168 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-09-14T05:31:28.357 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-09-14T05:31:28.496 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-14T05:31:28.560 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-fuse-2:19.3.0-4775.g7f24e589.el9.x86_64 2/2 2024-09-14T05:31:29.341 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4775.g7f24e589.el9.x86_64 2/2 2024-09-14T05:31:29.341 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-fuse-2:19.3.0-4775.g7f24e589.el9.x86_64 1/2 2024-09-14T05:31:29.609 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-14T05:31:29.609 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:31:29.609 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-09-14T05:31:29.610 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-fuse-2:19.3.0-4775.g7f24e589.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-14T05:31:29.610 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:31:29.610 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-09-14T05:31:29.712 INFO:teuthology.orchestra.run.smithi191.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 6.5 MB/s | 16 MB 00:02 2024-09-14T05:31:29.713 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-09-14T05:31:29.713 INFO:teuthology.orchestra.run.smithi191.stdout:Total 8.6 MB/s | 38 MB 00:04 2024-09-14T05:31:29.879 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph-volume 2024-09-14T05:31:30.069 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-09-14T05:31:30.095 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-09-14T05:31:30.095 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-09-14T05:31:30.456 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:01:35 ago on Sat 14 Sep 2024 05:29:55 AM UTC. 2024-09-14T05:31:30.532 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-09-14T05:31:30.532 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-09-14T05:31:30.592 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-09-14T05:31:30.592 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-09-14T05:31:30.593 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repository Size 2024-09-14T05:31:30.593 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-09-14T05:31:30.593 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-09-14T05:31:30.593 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-volume noarch 2:19.3.0-4775.g7f24e589.el9 ceph-noarch 285 k 2024-09-14T05:31:30.593 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:31:30.593 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-09-14T05:31:30.593 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-09-14T05:31:30.593 INFO:teuthology.orchestra.run.smithi143.stdout:Install 1 Package 2024-09-14T05:31:30.593 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:31:30.593 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 285 k 2024-09-14T05:31:30.593 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 1.3 M 2024-09-14T05:31:30.593 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-09-14T05:31:30.800 INFO:teuthology.orchestra.run.smithi143.stdout:ceph-volume-19.3.0-4775.g7f24e589.el9.noarch.rp 1.4 MB/s | 285 kB 00:00 2024-09-14T05:31:30.800 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-09-14T05:31:30.801 INFO:teuthology.orchestra.run.smithi143.stdout:Total 1.3 MB/s | 285 kB 00:00 2024-09-14T05:31:30.801 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-09-14T05:31:30.809 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-09-14T05:31:30.809 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-09-14T05:31:30.841 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-09-14T05:31:30.841 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-09-14T05:31:31.061 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-09-14T05:31:31.144 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-volume-2:19.3.0-4775.g7f24e589.el9.noarch 1/1 2024-09-14T05:31:31.160 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-volume-2:19.3.0-4775.g7f24e589.el9.noarch 1/1 2024-09-14T05:31:31.160 INFO:teuthology.orchestra.run.smithi143.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T05:31:31.160 INFO:teuthology.orchestra.run.smithi143.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-14T05:31:31.160 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:31:31.292 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-09-14T05:31:31.369 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-14T05:31:31.410 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-14T05:31:31.508 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-14T05:31:31.544 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-14T05:31:31.728 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-14T05:31:31.752 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-14T05:31:31.827 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-14T05:31:32.120 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-volume-2:19.3.0-4775.g7f24e589.el9.noarch 1/1 2024-09-14T05:31:32.120 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:31:32.120 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-09-14T05:31:32.120 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-volume-2:19.3.0-4775.g7f24e589.el9.noarch 2024-09-14T05:31:32.121 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:31:32.121 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-09-14T05:31:32.420 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install librados-devel 2024-09-14T05:31:32.491 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-14T05:31:32.606 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-14T05:31:33.000 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:01:37 ago on Sat 14 Sep 2024 05:29:55 AM UTC. 2024-09-14T05:31:33.136 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-09-14T05:31:33.137 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-09-14T05:31:33.137 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repo Size 2024-09-14T05:31:33.137 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-09-14T05:31:33.137 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-09-14T05:31:33.137 INFO:teuthology.orchestra.run.smithi143.stdout: librados-devel x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 126 k 2024-09-14T05:31:33.137 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:31:33.137 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-09-14T05:31:33.137 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-09-14T05:31:33.137 INFO:teuthology.orchestra.run.smithi143.stdout:Install 1 Package 2024-09-14T05:31:33.137 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:31:33.138 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 126 k 2024-09-14T05:31:33.138 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 456 k 2024-09-14T05:31:33.138 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-09-14T05:31:33.339 INFO:teuthology.orchestra.run.smithi143.stdout:librados-devel-19.3.0-4775.g7f24e589.el9.x86_64 629 kB/s | 126 kB 00:00 2024-09-14T05:31:33.340 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-09-14T05:31:33.340 INFO:teuthology.orchestra.run.smithi143.stdout:Total 625 kB/s | 126 kB 00:00 2024-09-14T05:31:33.340 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-09-14T05:31:33.345 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-09-14T05:31:33.345 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-09-14T05:31:33.410 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-09-14T05:31:33.410 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-09-14T05:31:33.559 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-09-14T05:31:33.661 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : librados-devel-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T05:31:33.838 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-14T05:31:34.069 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: librados-devel-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T05:31:34.329 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librados-devel-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T05:31:34.330 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:31:34.330 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-09-14T05:31:34.330 INFO:teuthology.orchestra.run.smithi143.stdout: librados-devel-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T05:31:34.330 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:31:34.330 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-09-14T05:31:34.589 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install libcephfs2 2024-09-14T05:31:34.873 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-14T05:31:34.909 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4775.g7f24e 12/12 2024-09-14T05:31:35.163 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:01:40 ago on Sat 14 Sep 2024 05:29:55 AM UTC. 2024-09-14T05:31:35.254 INFO:teuthology.orchestra.run.smithi143.stdout:Package libcephfs2-2:19.3.0-4775.g7f24e589.el9.x86_64 is already installed. 2024-09-14T05:31:35.300 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-09-14T05:31:35.301 INFO:teuthology.orchestra.run.smithi143.stdout:Nothing to do. 2024-09-14T05:31:35.301 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-09-14T05:31:35.363 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install libcephfs-devel 2024-09-14T05:31:35.934 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:01:40 ago on Sat 14 Sep 2024 05:29:55 AM UTC. 2024-09-14T05:31:36.069 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-09-14T05:31:36.070 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-09-14T05:31:36.070 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repo Size 2024-09-14T05:31:36.070 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-09-14T05:31:36.070 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-09-14T05:31:36.070 INFO:teuthology.orchestra.run.smithi143.stdout: libcephfs-devel x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 32 k 2024-09-14T05:31:36.070 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:31:36.070 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-09-14T05:31:36.070 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-09-14T05:31:36.071 INFO:teuthology.orchestra.run.smithi143.stdout:Install 1 Package 2024-09-14T05:31:36.071 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:31:36.071 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 32 k 2024-09-14T05:31:36.071 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 142 k 2024-09-14T05:31:36.071 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-09-14T05:31:36.226 INFO:teuthology.orchestra.run.smithi143.stdout:libcephfs-devel-19.3.0-4775.g7f24e589.el9.x86_6 204 kB/s | 32 kB 00:00 2024-09-14T05:31:36.226 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-09-14T05:31:36.226 INFO:teuthology.orchestra.run.smithi143.stdout:Total 203 kB/s | 32 kB 00:00 2024-09-14T05:31:36.227 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-09-14T05:31:36.230 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-09-14T05:31:36.230 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-09-14T05:31:36.251 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-09-14T05:31:36.251 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-09-14T05:31:36.340 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-09-14T05:31:36.456 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libcephfs-devel-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T05:31:36.830 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T05:31:37.065 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libcephfs-devel-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T05:31:37.065 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:31:37.065 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-09-14T05:31:37.065 INFO:teuthology.orchestra.run.smithi143.stdout: libcephfs-devel-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T05:31:37.065 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:31:37.065 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-09-14T05:31:37.249 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install librados2 2024-09-14T05:31:37.489 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4775.g7f24e 12/12 2024-09-14T05:31:37.489 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4775.g7f24e 1/12 2024-09-14T05:31:37.489 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-14T05:31:37.489 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-14T05:31:37.489 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-14T05:31:37.489 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-14T05:31:37.489 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-14T05:31:37.489 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-14T05:31:37.490 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-14T05:31:37.490 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-14T05:31:37.490 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-14T05:31:37.490 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-14T05:31:37.815 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:01:42 ago on Sat 14 Sep 2024 05:29:55 AM UTC. 2024-09-14T05:31:37.876 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-14T05:31:37.876 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:31:37.876 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-09-14T05:31:37.876 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4775.g7f24e589.el9.noarch 2024-09-14T05:31:37.876 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-14T05:31:37.876 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-14T05:31:37.876 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-14T05:31:37.876 INFO:teuthology.orchestra.run.smithi191.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-14T05:31:37.876 INFO:teuthology.orchestra.run.smithi191.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-14T05:31:37.876 INFO:teuthology.orchestra.run.smithi191.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-14T05:31:37.876 INFO:teuthology.orchestra.run.smithi191.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-14T05:31:37.877 INFO:teuthology.orchestra.run.smithi191.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-14T05:31:37.877 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-14T05:31:37.877 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-14T05:31:37.877 INFO:teuthology.orchestra.run.smithi191.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-14T05:31:37.877 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:31:37.877 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-14T05:31:37.905 INFO:teuthology.orchestra.run.smithi143.stdout:Package librados2-2:19.3.0-4775.g7f24e589.el9.x86_64 is already installed. 2024-09-14T05:31:37.949 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-09-14T05:31:37.950 INFO:teuthology.orchestra.run.smithi143.stdout:Nothing to do. 2024-09-14T05:31:37.950 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-09-14T05:31:38.014 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install librbd1 2024-09-14T05:31:38.245 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-mgr-rook 2024-09-14T05:31:38.578 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:01:43 ago on Sat 14 Sep 2024 05:29:55 AM UTC. 2024-09-14T05:31:38.672 INFO:teuthology.orchestra.run.smithi143.stdout:Package librbd1-2:19.3.0-4775.g7f24e589.el9.x86_64 is already installed. 2024-09-14T05:31:38.715 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-09-14T05:31:38.716 INFO:teuthology.orchestra.run.smithi143.stdout:Nothing to do. 2024-09-14T05:31:38.716 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-09-14T05:31:38.781 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install python3-rados 2024-09-14T05:31:38.828 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:24 ago on Sat 14 Sep 2024 05:30:14 AM UTC. 2024-09-14T05:31:38.961 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-14T05:31:38.962 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-14T05:31:38.962 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-09-14T05:31:38.962 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-14T05:31:38.963 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-09-14T05:31:38.963 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-rook noarch 2:19.3.0-4775.g7f24e589.el9 ceph-noarch 49 k 2024-09-14T05:31:38.963 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-09-14T05:31:38.963 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-14T05:31:38.963 INFO:teuthology.orchestra.run.smithi191.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-14T05:31:38.963 INFO:teuthology.orchestra.run.smithi191.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-14T05:31:38.963 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-14T05:31:38.963 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-14T05:31:38.963 INFO:teuthology.orchestra.run.smithi191.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-14T05:31:38.963 INFO:teuthology.orchestra.run.smithi191.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-14T05:31:38.963 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-14T05:31:38.963 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-14T05:31:38.963 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-14T05:31:38.964 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-14T05:31:38.964 INFO:teuthology.orchestra.run.smithi191.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-14T05:31:38.964 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:31:38.964 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-09-14T05:31:38.964 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-14T05:31:38.964 INFO:teuthology.orchestra.run.smithi191.stdout:Install 13 Packages 2024-09-14T05:31:38.964 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:31:38.965 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 2.2 M 2024-09-14T05:31:38.965 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 26 M 2024-09-14T05:31:38.965 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-09-14T05:31:39.208 INFO:teuthology.orchestra.run.smithi191.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 350 kB/s | 26 kB 00:00 2024-09-14T05:31:39.233 INFO:teuthology.orchestra.run.smithi191.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 195 kB/s | 19 kB 00:00 2024-09-14T05:31:39.308 INFO:teuthology.orchestra.run.smithi191.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.9 MB/s | 222 kB 00:00 2024-09-14T05:31:39.334 INFO:teuthology.orchestra.run.smithi191.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.6 MB/s | 159 kB 00:00 2024-09-14T05:31:39.348 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:01:44 ago on Sat 14 Sep 2024 05:29:55 AM UTC. 2024-09-14T05:31:39.359 INFO:teuthology.orchestra.run.smithi191.stdout:(5/13): ceph-mgr-rook-19.3.0-4775.g7f24e589.el9 220 kB/s | 49 kB 00:00 2024-09-14T05:31:39.384 INFO:teuthology.orchestra.run.smithi191.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-14T05:31:39.409 INFO:teuthology.orchestra.run.smithi191.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.7 MB/s | 279 kB 00:00 2024-09-14T05:31:39.439 INFO:teuthology.orchestra.run.smithi143.stdout:Package python3-rados-2:19.3.0-4775.g7f24e589.el9.x86_64 is already installed. 2024-09-14T05:31:39.451 INFO:teuthology.orchestra.run.smithi191.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 349 kB/s | 32 kB 00:00 2024-09-14T05:31:39.477 INFO:teuthology.orchestra.run.smithi191.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 153 kB/s | 14 kB 00:00 2024-09-14T05:31:39.482 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-09-14T05:31:39.484 INFO:teuthology.orchestra.run.smithi143.stdout:Nothing to do. 2024-09-14T05:31:39.484 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-09-14T05:31:39.502 INFO:teuthology.orchestra.run.smithi191.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-14T05:31:39.528 INFO:teuthology.orchestra.run.smithi191.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 2.1 MB/s | 220 kB 00:00 2024-09-14T05:31:39.547 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install python3-rgw 2024-09-14T05:31:39.552 INFO:teuthology.orchestra.run.smithi191.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-09-14T05:31:39.595 INFO:teuthology.orchestra.run.smithi191.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.2 MB/s | 1.0 MB 00:00 2024-09-14T05:31:39.597 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-09-14T05:31:39.597 INFO:teuthology.orchestra.run.smithi191.stdout:Total 3.5 MB/s | 2.2 MB 00:00 2024-09-14T05:31:39.662 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-09-14T05:31:39.677 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-09-14T05:31:39.677 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-09-14T05:31:39.767 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-09-14T05:31:39.769 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-09-14T05:31:40.114 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:01:45 ago on Sat 14 Sep 2024 05:29:55 AM UTC. 2024-09-14T05:31:40.203 INFO:teuthology.orchestra.run.smithi143.stdout:Package python3-rgw-2:19.3.0-4775.g7f24e589.el9.x86_64 is already installed. 2024-09-14T05:31:40.247 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-09-14T05:31:40.247 INFO:teuthology.orchestra.run.smithi143.stdout:Nothing to do. 2024-09-14T05:31:40.248 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-09-14T05:31:40.270 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-09-14T05:31:40.316 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install python3-cephfs 2024-09-14T05:31:40.438 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-14T05:31:40.504 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-14T05:31:40.568 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-14T05:31:40.605 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-14T05:31:40.655 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-14T05:31:40.766 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-14T05:31:40.875 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-14T05:31:40.885 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:01:45 ago on Sat 14 Sep 2024 05:29:55 AM UTC. 2024-09-14T05:31:40.957 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-14T05:31:40.977 INFO:teuthology.orchestra.run.smithi143.stdout:Package python3-cephfs-2:19.3.0-4775.g7f24e589.el9.x86_64 is already installed. 2024-09-14T05:31:41.021 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-09-14T05:31:41.022 INFO:teuthology.orchestra.run.smithi143.stdout:Nothing to do. 2024-09-14T05:31:41.022 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-09-14T05:31:41.088 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install python3-rbd 2024-09-14T05:31:41.663 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:01:46 ago on Sat 14 Sep 2024 05:29:55 AM UTC. 2024-09-14T05:31:41.675 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-14T05:31:41.720 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-14T05:31:41.755 INFO:teuthology.orchestra.run.smithi143.stdout:Package python3-rbd-2:19.3.0-4775.g7f24e589.el9.x86_64 is already installed. 2024-09-14T05:31:41.762 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-14T05:31:41.801 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-09-14T05:31:41.802 INFO:teuthology.orchestra.run.smithi143.stdout:Nothing to do. 2024-09-14T05:31:41.802 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-09-14T05:31:41.810 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-14T05:31:41.832 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-rook-2:19.3.0-4775.g7f24e589.el9.noarch 13/13 2024-09-14T05:31:41.878 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install rbd-fuse 2024-09-14T05:31:42.464 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:01:47 ago on Sat 14 Sep 2024 05:29:55 AM UTC. 2024-09-14T05:31:42.600 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-09-14T05:31:42.601 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-09-14T05:31:42.601 INFO:teuthology.orchestra.run.smithi143.stdout: Package Architecture Version Repository Size 2024-09-14T05:31:42.601 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-09-14T05:31:42.601 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-09-14T05:31:42.601 INFO:teuthology.orchestra.run.smithi143.stdout: rbd-fuse x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 85 k 2024-09-14T05:31:42.601 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:31:42.601 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-09-14T05:31:42.601 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-09-14T05:31:42.602 INFO:teuthology.orchestra.run.smithi143.stdout:Install 1 Package 2024-09-14T05:31:42.602 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:31:42.602 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 85 k 2024-09-14T05:31:42.602 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 231 k 2024-09-14T05:31:42.602 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-09-14T05:31:42.668 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4775.g7f24e589.el9.noarch 13/13 2024-09-14T05:31:42.668 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4775.g7f24e589.el9.noarch 1/13 2024-09-14T05:31:42.668 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-14T05:31:42.668 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-14T05:31:42.668 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-14T05:31:42.668 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-14T05:31:42.669 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-14T05:31:42.669 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-14T05:31:42.669 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-14T05:31:42.669 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-14T05:31:42.669 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-14T05:31:42.669 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-14T05:31:42.669 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-14T05:31:42.796 INFO:teuthology.orchestra.run.smithi143.stdout:rbd-fuse-19.3.0-4775.g7f24e589.el9.x86_64.rpm 439 kB/s | 85 kB 00:00 2024-09-14T05:31:42.797 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-09-14T05:31:42.797 INFO:teuthology.orchestra.run.smithi143.stdout:Total 434 kB/s | 85 kB 00:00 2024-09-14T05:31:42.797 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-09-14T05:31:42.804 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-09-14T05:31:42.804 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-09-14T05:31:42.867 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-09-14T05:31:42.867 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-09-14T05:31:43.011 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-09-14T05:31:43.022 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-14T05:31:43.023 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:31:43.023 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-09-14T05:31:43.023 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-rook-2:19.3.0-4775.g7f24e589.el9.noarch 2024-09-14T05:31:43.023 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-14T05:31:43.023 INFO:teuthology.orchestra.run.smithi191.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-14T05:31:43.023 INFO:teuthology.orchestra.run.smithi191.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-14T05:31:43.023 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-14T05:31:43.023 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-14T05:31:43.023 INFO:teuthology.orchestra.run.smithi191.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-14T05:31:43.023 INFO:teuthology.orchestra.run.smithi191.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-14T05:31:43.023 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-14T05:31:43.023 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-14T05:31:43.023 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-14T05:31:43.023 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-14T05:31:43.024 INFO:teuthology.orchestra.run.smithi191.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-14T05:31:43.024 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:31:43.024 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-14T05:31:43.119 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : rbd-fuse-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T05:31:43.299 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-mgr-cephadm 2024-09-14T05:31:43.486 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T05:31:43.739 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : rbd-fuse-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T05:31:43.739 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:31:43.740 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-09-14T05:31:43.740 INFO:teuthology.orchestra.run.smithi143.stdout: rbd-fuse-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T05:31:43.740 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:31:43.740 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-09-14T05:31:43.867 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:29 ago on Sat 14 Sep 2024 05:30:14 AM UTC. 2024-09-14T05:31:43.973 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install rbd-mirror 2024-09-14T05:31:44.001 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-14T05:31:44.002 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-14T05:31:44.003 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-09-14T05:31:44.003 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-14T05:31:44.003 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-09-14T05:31:44.003 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4775.g7f24e589.el9 ceph-noarch 154 k 2024-09-14T05:31:44.003 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-09-14T05:31:44.003 INFO:teuthology.orchestra.run.smithi191.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-14T05:31:44.003 INFO:teuthology.orchestra.run.smithi191.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-14T05:31:44.003 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-14T05:31:44.003 INFO:teuthology.orchestra.run.smithi191.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-14T05:31:44.003 INFO:teuthology.orchestra.run.smithi191.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-14T05:31:44.003 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:31:44.003 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-09-14T05:31:44.003 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-14T05:31:44.004 INFO:teuthology.orchestra.run.smithi191.stdout:Install 6 Packages 2024-09-14T05:31:44.004 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:31:44.004 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 7.0 M 2024-09-14T05:31:44.004 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 33 M 2024-09-14T05:31:44.004 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-09-14T05:31:44.330 INFO:teuthology.orchestra.run.smithi191.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.9 MB/s | 248 kB 00:00 2024-09-14T05:31:44.388 INFO:teuthology.orchestra.run.smithi191.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 9.2 MB/s | 548 kB 00:00 2024-09-14T05:31:44.413 INFO:teuthology.orchestra.run.smithi191.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-09-14T05:31:44.439 INFO:teuthology.orchestra.run.smithi191.stdout:(4/6): ceph-mgr-cephadm-19.3.0-4775.g7f24e589.e 649 kB/s | 154 kB 00:00 2024-09-14T05:31:44.464 INFO:teuthology.orchestra.run.smithi191.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.4 MB/s | 74 kB 00:00 2024-09-14T05:31:44.544 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:01:49 ago on Sat 14 Sep 2024 05:29:55 AM UTC. 2024-09-14T05:31:44.581 INFO:teuthology.orchestra.run.smithi191.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 6.0 MB 00:00 2024-09-14T05:31:44.583 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-09-14T05:31:44.583 INFO:teuthology.orchestra.run.smithi191.stdout:Total 12 MB/s | 7.0 MB 00:00 2024-09-14T05:31:44.653 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-09-14T05:31:44.664 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-09-14T05:31:44.665 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-09-14T05:31:44.679 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-09-14T05:31:44.680 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-09-14T05:31:44.680 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repo Size 2024-09-14T05:31:44.680 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-09-14T05:31:44.680 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-09-14T05:31:44.680 INFO:teuthology.orchestra.run.smithi143.stdout: rbd-mirror x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 3.2 M 2024-09-14T05:31:44.680 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:31:44.680 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-09-14T05:31:44.680 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-09-14T05:31:44.680 INFO:teuthology.orchestra.run.smithi143.stdout:Install 1 Package 2024-09-14T05:31:44.680 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:31:44.680 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 3.2 M 2024-09-14T05:31:44.681 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 14 M 2024-09-14T05:31:44.681 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-09-14T05:31:44.765 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-09-14T05:31:44.767 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-09-14T05:31:45.089 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-09-14T05:31:45.262 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-14T05:31:45.316 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-14T05:31:45.359 INFO:teuthology.orchestra.run.smithi143.stdout:rbd-mirror-19.3.0-4775.g7f24e589.el9.x86_64.rpm 4.7 MB/s | 3.2 MB 00:00 2024-09-14T05:31:45.359 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-09-14T05:31:45.360 INFO:teuthology.orchestra.run.smithi143.stdout:Total 4.7 MB/s | 3.2 MB 00:00 2024-09-14T05:31:45.360 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-09-14T05:31:45.369 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-09-14T05:31:45.369 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-09-14T05:31:45.452 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-09-14T05:31:45.452 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-09-14T05:31:45.831 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-14T05:31:45.884 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-09-14T05:31:45.918 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-14T05:31:45.943 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : rbd-mirror-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T05:31:45.969 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T05:31:45.969 INFO:teuthology.orchestra.run.smithi143.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T05:31:45.969 INFO:teuthology.orchestra.run.smithi143.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-14T05:31:45.969 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-14T05:31:45.969 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-14T05:31:45.969 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:31:45.986 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-14T05:31:46.011 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4775.g7f24e589.el9.noarch 6/6 2024-09-14T05:31:46.854 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4775.g7f24e589.el9.noarch 6/6 2024-09-14T05:31:46.854 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4775.g7f24e589.el9.noarch 1/6 2024-09-14T05:31:46.854 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-14T05:31:46.855 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-14T05:31:46.855 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-14T05:31:46.855 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-14T05:31:47.055 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : rbd-mirror-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T05:31:47.055 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:31:47.055 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-09-14T05:31:47.055 INFO:teuthology.orchestra.run.smithi143.stdout: rbd-mirror-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T05:31:47.055 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:31:47.055 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-09-14T05:31:47.158 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-14T05:31:47.158 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:31:47.158 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-09-14T05:31:47.158 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-cephadm-2:19.3.0-4775.g7f24e589.el9.noarch 2024-09-14T05:31:47.158 INFO:teuthology.orchestra.run.smithi191.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-14T05:31:47.158 INFO:teuthology.orchestra.run.smithi191.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-14T05:31:47.158 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-14T05:31:47.158 INFO:teuthology.orchestra.run.smithi191.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-14T05:31:47.159 INFO:teuthology.orchestra.run.smithi191.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-14T05:31:47.159 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:31:47.159 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-14T05:31:47.286 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install rbd-nbd 2024-09-14T05:31:47.444 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-fuse 2024-09-14T05:31:47.869 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:01:52 ago on Sat 14 Sep 2024 05:29:55 AM UTC. 2024-09-14T05:31:48.005 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-09-14T05:31:48.005 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-09-14T05:31:48.006 INFO:teuthology.orchestra.run.smithi143.stdout: Package Architecture Version Repository Size 2024-09-14T05:31:48.006 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-09-14T05:31:48.006 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-09-14T05:31:48.006 INFO:teuthology.orchestra.run.smithi143.stdout: rbd-nbd x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 171 k 2024-09-14T05:31:48.006 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:31:48.006 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-09-14T05:31:48.006 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-09-14T05:31:48.006 INFO:teuthology.orchestra.run.smithi143.stdout:Install 1 Package 2024-09-14T05:31:48.006 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:31:48.006 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 171 k 2024-09-14T05:31:48.006 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 490 k 2024-09-14T05:31:48.006 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-09-14T05:31:48.013 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:34 ago on Sat 14 Sep 2024 05:30:14 AM UTC. 2024-09-14T05:31:48.146 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-14T05:31:48.147 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-14T05:31:48.147 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-09-14T05:31:48.147 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-14T05:31:48.147 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-09-14T05:31:48.147 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-fuse x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 900 k 2024-09-14T05:31:48.147 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-09-14T05:31:48.147 INFO:teuthology.orchestra.run.smithi191.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-14T05:31:48.147 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:31:48.148 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-09-14T05:31:48.148 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-14T05:31:48.148 INFO:teuthology.orchestra.run.smithi191.stdout:Install 2 Packages 2024-09-14T05:31:48.148 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:31:48.148 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 979 k 2024-09-14T05:31:48.148 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 2.9 M 2024-09-14T05:31:48.149 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-09-14T05:31:48.204 INFO:teuthology.orchestra.run.smithi143.stdout:rbd-nbd-19.3.0-4775.g7f24e589.el9.x86_64.rpm 865 kB/s | 171 kB 00:00 2024-09-14T05:31:48.205 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-09-14T05:31:48.205 INFO:teuthology.orchestra.run.smithi143.stdout:Total 858 kB/s | 171 kB 00:00 2024-09-14T05:31:48.205 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-09-14T05:31:48.213 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-09-14T05:31:48.213 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-09-14T05:31:48.276 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-09-14T05:31:48.277 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-09-14T05:31:48.445 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-09-14T05:31:48.527 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : rbd-nbd-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T05:31:48.533 INFO:teuthology.orchestra.run.smithi191.stdout:(1/2): ceph-fuse-19.3.0-4775.g7f24e589.el9.x86_ 3.4 MB/s | 900 kB 00:00 2024-09-14T05:31:48.909 INFO:teuthology.orchestra.run.smithi191.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 125 kB/s | 80 kB 00:00 2024-09-14T05:31:48.909 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-09-14T05:31:48.910 INFO:teuthology.orchestra.run.smithi191.stdout:Total 1.3 MB/s | 979 kB 00:00 2024-09-14T05:31:48.920 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T05:31:48.922 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-09-14T05:31:48.944 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-09-14T05:31:48.945 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-09-14T05:31:49.026 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-09-14T05:31:49.026 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-09-14T05:31:49.185 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : rbd-nbd-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T05:31:49.185 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:31:49.185 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-09-14T05:31:49.185 INFO:teuthology.orchestra.run.smithi143.stdout: rbd-nbd-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T05:31:49.185 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:31:49.185 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-09-14T05:31:49.216 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-09-14T05:31:49.353 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-14T05:31:49.425 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-fuse-2:19.3.0-4775.g7f24e589.el9.x86_64 2/2 2024-09-14T05:31:49.431 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install rbd-mirror 2024-09-14T05:31:50.003 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:01:55 ago on Sat 14 Sep 2024 05:29:55 AM UTC. 2024-09-14T05:31:50.095 INFO:teuthology.orchestra.run.smithi143.stdout:Package rbd-mirror-2:19.3.0-4775.g7f24e589.el9.x86_64 is already installed. 2024-09-14T05:31:50.126 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4775.g7f24e589.el9.x86_64 2/2 2024-09-14T05:31:50.126 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-fuse-2:19.3.0-4775.g7f24e589.el9.x86_64 1/2 2024-09-14T05:31:50.139 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-09-14T05:31:50.140 INFO:teuthology.orchestra.run.smithi143.stdout:Nothing to do. 2024-09-14T05:31:50.140 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-09-14T05:31:50.205 DEBUG:teuthology.parallel:result is None 2024-09-14T05:31:50.426 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-14T05:31:50.427 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:31:50.427 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-09-14T05:31:50.427 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-fuse-2:19.3.0-4775.g7f24e589.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-14T05:31:50.427 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:31:50.427 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-14T05:31:50.688 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-volume 2024-09-14T05:31:51.276 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:37 ago on Sat 14 Sep 2024 05:30:14 AM UTC. 2024-09-14T05:31:51.412 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-14T05:31:51.413 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-14T05:31:51.413 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-09-14T05:31:51.413 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-14T05:31:51.413 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-09-14T05:31:51.413 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-volume noarch 2:19.3.0-4775.g7f24e589.el9 ceph-noarch 285 k 2024-09-14T05:31:51.413 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:31:51.413 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-09-14T05:31:51.414 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-14T05:31:51.414 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-09-14T05:31:51.414 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:31:51.414 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 285 k 2024-09-14T05:31:51.414 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 1.3 M 2024-09-14T05:31:51.414 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-09-14T05:31:51.621 INFO:teuthology.orchestra.run.smithi191.stdout:ceph-volume-19.3.0-4775.g7f24e589.el9.noarch.rp 1.3 MB/s | 285 kB 00:00 2024-09-14T05:31:51.622 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-09-14T05:31:51.622 INFO:teuthology.orchestra.run.smithi191.stdout:Total 1.3 MB/s | 285 kB 00:00 2024-09-14T05:31:51.622 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-09-14T05:31:51.630 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-09-14T05:31:51.631 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-09-14T05:31:51.656 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-09-14T05:31:51.657 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-09-14T05:31:51.872 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-09-14T05:31:51.948 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-volume-2:19.3.0-4775.g7f24e589.el9.noarch 1/1 2024-09-14T05:31:51.965 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-volume-2:19.3.0-4775.g7f24e589.el9.noarch 1/1 2024-09-14T05:31:51.965 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T05:31:51.965 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-14T05:31:51.965 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:31:53.023 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-volume-2:19.3.0-4775.g7f24e589.el9.noarch 1/1 2024-09-14T05:31:53.024 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:31:53.024 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-09-14T05:31:53.024 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-volume-2:19.3.0-4775.g7f24e589.el9.noarch 2024-09-14T05:31:53.024 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:31:53.024 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-14T05:31:53.397 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install librados-devel 2024-09-14T05:31:53.977 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:39 ago on Sat 14 Sep 2024 05:30:14 AM UTC. 2024-09-14T05:31:54.113 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-14T05:31:54.114 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-14T05:31:54.114 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repo Size 2024-09-14T05:31:54.114 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-14T05:31:54.114 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-09-14T05:31:54.114 INFO:teuthology.orchestra.run.smithi191.stdout: librados-devel x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 126 k 2024-09-14T05:31:54.114 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:31:54.115 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-09-14T05:31:54.115 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-14T05:31:54.115 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-09-14T05:31:54.115 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:31:54.115 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 126 k 2024-09-14T05:31:54.115 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 456 k 2024-09-14T05:31:54.115 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-09-14T05:31:54.335 INFO:teuthology.orchestra.run.smithi191.stdout:librados-devel-19.3.0-4775.g7f24e589.el9.x86_64 576 kB/s | 126 kB 00:00 2024-09-14T05:31:54.335 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-09-14T05:31:54.336 INFO:teuthology.orchestra.run.smithi191.stdout:Total 572 kB/s | 126 kB 00:00 2024-09-14T05:31:54.336 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-09-14T05:31:54.341 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-09-14T05:31:54.342 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-09-14T05:31:54.405 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-09-14T05:31:54.406 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-09-14T05:31:54.554 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-09-14T05:31:54.657 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librados-devel-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T05:31:55.117 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librados-devel-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T05:31:55.345 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados-devel-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T05:31:55.345 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:31:55.345 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-09-14T05:31:55.345 INFO:teuthology.orchestra.run.smithi191.stdout: librados-devel-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T05:31:55.345 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:31:55.345 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-14T05:31:55.578 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install libcephfs2 2024-09-14T05:31:56.152 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:42 ago on Sat 14 Sep 2024 05:30:14 AM UTC. 2024-09-14T05:31:56.243 INFO:teuthology.orchestra.run.smithi191.stdout:Package libcephfs2-2:19.3.0-4775.g7f24e589.el9.x86_64 is already installed. 2024-09-14T05:31:56.287 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-14T05:31:56.288 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-09-14T05:31:56.289 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-14T05:31:56.357 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install libcephfs-devel 2024-09-14T05:31:56.929 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:42 ago on Sat 14 Sep 2024 05:30:14 AM UTC. 2024-09-14T05:31:57.083 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-14T05:31:57.084 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-14T05:31:57.084 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repo Size 2024-09-14T05:31:57.084 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-14T05:31:57.084 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-09-14T05:31:57.085 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs-devel x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 32 k 2024-09-14T05:31:57.085 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:31:57.085 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-09-14T05:31:57.085 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-14T05:31:57.085 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-09-14T05:31:57.085 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:31:57.085 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 32 k 2024-09-14T05:31:57.085 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 142 k 2024-09-14T05:31:57.085 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-09-14T05:31:57.255 INFO:teuthology.orchestra.run.smithi191.stdout:libcephfs-devel-19.3.0-4775.g7f24e589.el9.x86_6 186 kB/s | 32 kB 00:00 2024-09-14T05:31:57.256 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-09-14T05:31:57.256 INFO:teuthology.orchestra.run.smithi191.stdout:Total 185 kB/s | 32 kB 00:00 2024-09-14T05:31:57.256 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-09-14T05:31:57.260 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-09-14T05:31:57.260 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-09-14T05:31:57.284 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-09-14T05:31:57.285 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-09-14T05:31:57.379 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-09-14T05:31:57.494 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephfs-devel-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T05:31:57.889 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T05:31:58.129 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephfs-devel-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T05:31:58.129 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:31:58.129 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-09-14T05:31:58.129 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs-devel-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T05:31:58.129 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:31:58.129 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-14T05:31:58.320 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install librados2 2024-09-14T05:31:58.878 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:44 ago on Sat 14 Sep 2024 05:30:14 AM UTC. 2024-09-14T05:31:58.969 INFO:teuthology.orchestra.run.smithi191.stdout:Package librados2-2:19.3.0-4775.g7f24e589.el9.x86_64 is already installed. 2024-09-14T05:31:59.012 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-14T05:31:59.013 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-09-14T05:31:59.013 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-14T05:31:59.079 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install librbd1 2024-09-14T05:31:59.647 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:45 ago on Sat 14 Sep 2024 05:30:14 AM UTC. 2024-09-14T05:31:59.739 INFO:teuthology.orchestra.run.smithi191.stdout:Package librbd1-2:19.3.0-4775.g7f24e589.el9.x86_64 is already installed. 2024-09-14T05:31:59.782 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-14T05:31:59.783 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-09-14T05:31:59.783 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-14T05:31:59.857 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install python3-rados 2024-09-14T05:32:00.418 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:46 ago on Sat 14 Sep 2024 05:30:14 AM UTC. 2024-09-14T05:32:00.509 INFO:teuthology.orchestra.run.smithi191.stdout:Package python3-rados-2:19.3.0-4775.g7f24e589.el9.x86_64 is already installed. 2024-09-14T05:32:00.553 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-14T05:32:00.554 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-09-14T05:32:00.554 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-14T05:32:00.622 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install python3-rgw 2024-09-14T05:32:01.185 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:47 ago on Sat 14 Sep 2024 05:30:14 AM UTC. 2024-09-14T05:32:01.275 INFO:teuthology.orchestra.run.smithi191.stdout:Package python3-rgw-2:19.3.0-4775.g7f24e589.el9.x86_64 is already installed. 2024-09-14T05:32:01.318 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-14T05:32:01.319 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-09-14T05:32:01.319 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-14T05:32:01.388 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install python3-cephfs 2024-09-14T05:32:01.963 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:47 ago on Sat 14 Sep 2024 05:30:14 AM UTC. 2024-09-14T05:32:02.055 INFO:teuthology.orchestra.run.smithi191.stdout:Package python3-cephfs-2:19.3.0-4775.g7f24e589.el9.x86_64 is already installed. 2024-09-14T05:32:02.098 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-14T05:32:02.099 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-09-14T05:32:02.099 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-14T05:32:02.162 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install python3-rbd 2024-09-14T05:32:02.726 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:48 ago on Sat 14 Sep 2024 05:30:14 AM UTC. 2024-09-14T05:32:02.816 INFO:teuthology.orchestra.run.smithi191.stdout:Package python3-rbd-2:19.3.0-4775.g7f24e589.el9.x86_64 is already installed. 2024-09-14T05:32:02.859 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-14T05:32:02.860 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-09-14T05:32:02.861 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-14T05:32:02.929 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install rbd-fuse 2024-09-14T05:32:03.493 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:49 ago on Sat 14 Sep 2024 05:30:14 AM UTC. 2024-09-14T05:32:03.627 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-14T05:32:03.628 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-14T05:32:03.628 INFO:teuthology.orchestra.run.smithi191.stdout: Package Architecture Version Repository Size 2024-09-14T05:32:03.628 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-14T05:32:03.628 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-09-14T05:32:03.628 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-fuse x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 85 k 2024-09-14T05:32:03.628 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:03.628 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-09-14T05:32:03.628 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-14T05:32:03.629 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-09-14T05:32:03.629 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:03.629 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 85 k 2024-09-14T05:32:03.629 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 231 k 2024-09-14T05:32:03.629 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-09-14T05:32:03.817 INFO:teuthology.orchestra.run.smithi191.stdout:rbd-fuse-19.3.0-4775.g7f24e589.el9.x86_64.rpm 452 kB/s | 85 kB 00:00 2024-09-14T05:32:03.817 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-09-14T05:32:03.818 INFO:teuthology.orchestra.run.smithi191.stdout:Total 449 kB/s | 85 kB 00:00 2024-09-14T05:32:03.818 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-09-14T05:32:03.824 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-09-14T05:32:03.824 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-09-14T05:32:03.888 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-09-14T05:32:03.888 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-09-14T05:32:04.032 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-09-14T05:32:04.131 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-fuse-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T05:32:04.539 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T05:32:04.804 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-fuse-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T05:32:04.804 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:04.804 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-09-14T05:32:04.804 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-fuse-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T05:32:04.804 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:04.804 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-14T05:32:05.059 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install rbd-mirror 2024-09-14T05:32:05.633 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:51 ago on Sat 14 Sep 2024 05:30:14 AM UTC. 2024-09-14T05:32:05.769 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-14T05:32:05.769 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-14T05:32:05.769 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repo Size 2024-09-14T05:32:05.770 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-14T05:32:05.770 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-09-14T05:32:05.770 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-mirror x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 3.2 M 2024-09-14T05:32:05.770 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:05.770 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-09-14T05:32:05.770 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-14T05:32:05.770 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-09-14T05:32:05.770 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:05.770 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 3.2 M 2024-09-14T05:32:05.770 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 14 M 2024-09-14T05:32:05.770 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-09-14T05:32:06.094 INFO:teuthology.orchestra.run.smithi191.stdout:rbd-mirror-19.3.0-4775.g7f24e589.el9.x86_64.rpm 9.8 MB/s | 3.2 MB 00:00 2024-09-14T05:32:06.095 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-09-14T05:32:06.095 INFO:teuthology.orchestra.run.smithi191.stdout:Total 9.8 MB/s | 3.2 MB 00:00 2024-09-14T05:32:06.095 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-09-14T05:32:06.105 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-09-14T05:32:06.105 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-09-14T05:32:06.187 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-09-14T05:32:06.187 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-09-14T05:32:06.626 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-09-14T05:32:06.695 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-mirror-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T05:32:06.720 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T05:32:06.721 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-14T05:32:06.721 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-14T05:32:06.721 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-14T05:32:06.721 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-14T05:32:06.721 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:07.920 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-mirror-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T05:32:07.921 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:07.921 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-09-14T05:32:07.921 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-mirror-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T05:32:07.921 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:07.921 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-14T05:32:08.146 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install rbd-nbd 2024-09-14T05:32:08.727 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:54 ago on Sat 14 Sep 2024 05:30:14 AM UTC. 2024-09-14T05:32:08.862 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-14T05:32:08.863 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-14T05:32:08.863 INFO:teuthology.orchestra.run.smithi191.stdout: Package Architecture Version Repository Size 2024-09-14T05:32:08.863 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-14T05:32:08.863 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-09-14T05:32:08.863 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-nbd x86_64 2:19.3.0-4775.g7f24e589.el9 ceph 171 k 2024-09-14T05:32:08.863 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:08.864 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-09-14T05:32:08.864 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-14T05:32:08.864 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-09-14T05:32:08.864 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:08.864 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 171 k 2024-09-14T05:32:08.864 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 490 k 2024-09-14T05:32:08.864 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-09-14T05:32:09.076 INFO:teuthology.orchestra.run.smithi191.stdout:rbd-nbd-19.3.0-4775.g7f24e589.el9.x86_64.rpm 804 kB/s | 171 kB 00:00 2024-09-14T05:32:09.077 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-09-14T05:32:09.077 INFO:teuthology.orchestra.run.smithi191.stdout:Total 799 kB/s | 171 kB 00:00 2024-09-14T05:32:09.077 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-09-14T05:32:09.085 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-09-14T05:32:09.085 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-09-14T05:32:09.148 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-09-14T05:32:09.148 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-09-14T05:32:09.315 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-09-14T05:32:09.417 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-nbd-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T05:32:09.858 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T05:32:10.145 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-nbd-2:19.3.0-4775.g7f24e589.el9.x86_64 1/1 2024-09-14T05:32:10.145 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:10.145 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-09-14T05:32:10.145 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-nbd-2:19.3.0-4775.g7f24e589.el9.x86_64 2024-09-14T05:32:10.145 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:10.145 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-14T05:32:10.368 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install rbd-mirror 2024-09-14T05:32:10.950 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:56 ago on Sat 14 Sep 2024 05:30:14 AM UTC. 2024-09-14T05:32:11.043 INFO:teuthology.orchestra.run.smithi191.stdout:Package rbd-mirror-2:19.3.0-4775.g7f24e589.el9.x86_64 is already installed. 2024-09-14T05:32:11.088 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-14T05:32:11.089 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-09-14T05:32:11.089 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-14T05:32:11.170 DEBUG:teuthology.parallel:result is None 2024-09-14T05:32:11.171 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7f24e589328ae9e639c0009d93340e5b235d0a47 2024-09-14T05:32:11.314 DEBUG:teuthology.orchestra.run.smithi143:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-14T05:32:11.340 INFO:teuthology.orchestra.run.smithi143.stdout:19.3.0-4775.g7f24e589.el9 2024-09-14T05:32:11.341 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4775.g7f24e589.el9 2024-09-14T05:32:11.341 INFO:teuthology.task.install:The correct ceph version 19.3.0-4775.g7f24e589 is installed. 2024-09-14T05:32:11.343 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7f24e589328ae9e639c0009d93340e5b235d0a47 2024-09-14T05:32:11.482 DEBUG:teuthology.orchestra.run.smithi191:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-14T05:32:11.503 INFO:teuthology.orchestra.run.smithi191.stdout:19.3.0-4775.g7f24e589.el9 2024-09-14T05:32:11.503 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4775.g7f24e589.el9 2024-09-14T05:32:11.504 INFO:teuthology.task.install:The correct ceph version 19.3.0-4775.g7f24e589 is installed. 2024-09-14T05:32:11.505 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-14T05:32:11.506 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-14T05:32:11.506 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-14T05:32:11.537 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-14T05:32:11.537 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-14T05:32:11.570 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-14T05:32:11.571 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-14T05:32:11.571 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/usr/bin/daemon-helper 2024-09-14T05:32:11.605 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-14T05:32:11.674 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-14T05:32:11.674 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/daemon-helper 2024-09-14T05:32:11.699 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-14T05:32:11.764 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-14T05:32:11.765 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-14T05:32:11.765 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-14T05:32:11.793 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-14T05:32:11.860 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-14T05:32:11.860 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-14T05:32:11.885 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-14T05:32:11.949 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-14T05:32:11.949 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-14T05:32:11.949 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/usr/bin/stdin-killer 2024-09-14T05:32:11.975 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-14T05:32:12.041 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-14T05:32:12.041 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/stdin-killer 2024-09-14T05:32:12.066 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-14T05:32:12.130 INFO:teuthology.run_tasks:Running task ceph... 2024-09-14T05:32:12.215 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-14T05:32:12.215 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 777 /var/log/ceph 2024-09-14T05:32:12.220 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 777 /var/log/ceph 2024-09-14T05:32:12.245 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-14T05:32:12.245 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-14T05:32:12.284 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-14T05:32:12.310 INFO:tasks.ceph:Creating extra log directories... 2024-09-14T05:32:12.310 DEBUG:teuthology.orchestra.run.smithi143:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-14T05:32:12.350 DEBUG:teuthology.orchestra.run.smithi191:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-14T05:32:12.388 INFO:tasks.ceph:Creating ceph cluster cluster1... 2024-09-14T05:32:12.389 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-14T05:32:12.389 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-10_20:16:17-rbd-main-distro-default-smithi/7900324', 'branch': 'main', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zstd policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7900324', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'teuthology-2024-09-10_20:16:17-rbd-main-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '7f24e589328ae9e639c0009d93340e5b235d0a47'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '7f24e589328ae9e639c0009d93340e5b235d0a47'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '0d61375b8ab53be4c233f5b93e463edfdff9316a'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 950, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 5839, 'sha1': '7f24e589328ae9e639c0009d93340e5b235d0a47', 'sleep_before_teardown': 0, 'subset': '99/128', 'suite': 'rbd', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_0d61375b8ab53be4c233f5b93e463edfdff9316a/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '0d61375b8ab53be4c233f5b93e463edfdff9316a', 'targets': {'smithi143.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBKp8YEKbvZo4dBiDX4/t6S8Va2K3MQZi0ygsc4Dx7PCp5DriXj2KiaYHnovMOotaCtpOJyPkhsJB6V3toaXXbo=', 'smithi191.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL2v073v2G1I34ew/oySIIhRcoJk/PRN/EB4kIRX8Gp3X0OidBnpvoZx252VCP35d73k0EnPO2qJdEPgW68DolU='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '7f24e589328ae9e639c0009d93340e5b235d0a47'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '7f24e589328ae9e639c0009d93340e5b235d0a47'}}, {'ceph': {'cluster': 'cluster2'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering,exclusive-lock,object-map,fast-diff', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_MODE': 'snapshot', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-10_20:16:17', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-14T05:32:12.390 DEBUG:teuthology.orchestra.run.smithi143:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-09-14T05:32:12.417 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-09-14T05:32:12.432 DEBUG:teuthology.orchestra.run.smithi143:> sudo install -d -m0777 -- /var/run/ceph 2024-09-14T05:32:12.473 DEBUG:teuthology.orchestra.run.smithi191:> sudo install -d -m0777 -- /var/run/ceph 2024-09-14T05:32:12.498 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-14T05:32:12.498 DEBUG:teuthology.orchestra.run.smithi143:> dd if=/scratch_devs of=/dev/stdout 2024-09-14T05:32:12.553 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-14T05:32:12.554 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/vg_nvme/lv_1 2024-09-14T05:32:12.609 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-14T05:32:12.609 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-14T05:32:12.610 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-09-14T05:32:12.610 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-14T05:32:12.610 INFO:teuthology.orchestra.run.smithi143.stdout:Context: system_u:object_r:device_t:s0 2024-09-14T05:32:12.610 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2024-09-14 05:31:46.223929714 +0000 2024-09-14T05:32:12.610 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2024-09-14 05:29:16.677165113 +0000 2024-09-14T05:32:12.610 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2024-09-14 05:29:16.677165113 +0000 2024-09-14T05:32:12.610 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: 2024-09-14 05:29:16.677165113 +0000 2024-09-14T05:32:12.610 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-14T05:32:12.672 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2024-09-14T05:32:12.673 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2024-09-14T05:32:12.673 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.000206749 s, 2.5 MB/s 2024-09-14T05:32:12.674 DEBUG:teuthology.orchestra.run.smithi143:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-14T05:32:12.730 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/vg_nvme/lv_2 2024-09-14T05:32:12.786 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-14T05:32:12.786 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-14T05:32:12.786 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-09-14T05:32:12.786 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-14T05:32:12.786 INFO:teuthology.orchestra.run.smithi143.stdout:Context: system_u:object_r:device_t:s0 2024-09-14T05:32:12.786 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2024-09-14 05:31:46.223929714 +0000 2024-09-14T05:32:12.786 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2024-09-14 05:29:16.994156314 +0000 2024-09-14T05:32:12.786 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2024-09-14 05:29:16.994156314 +0000 2024-09-14T05:32:12.786 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: 2024-09-14 05:29:16.994156314 +0000 2024-09-14T05:32:12.787 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-14T05:32:12.849 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2024-09-14T05:32:12.849 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2024-09-14T05:32:12.849 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.000175183 s, 2.9 MB/s 2024-09-14T05:32:12.850 DEBUG:teuthology.orchestra.run.smithi143:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-14T05:32:12.907 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/vg_nvme/lv_3 2024-09-14T05:32:12.961 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-14T05:32:12.962 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-14T05:32:12.962 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-14T05:32:12.962 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-14T05:32:12.962 INFO:teuthology.orchestra.run.smithi143.stdout:Context: system_u:object_r:device_t:s0 2024-09-14T05:32:12.962 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2024-09-14 05:31:46.224929686 +0000 2024-09-14T05:32:12.962 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2024-09-14 05:29:17.320147265 +0000 2024-09-14T05:32:12.962 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2024-09-14 05:29:17.320147265 +0000 2024-09-14T05:32:12.962 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: 2024-09-14 05:29:17.320147265 +0000 2024-09-14T05:32:12.962 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-14T05:32:13.025 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2024-09-14T05:32:13.025 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2024-09-14T05:32:13.025 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.00018965 s, 2.7 MB/s 2024-09-14T05:32:13.026 DEBUG:teuthology.orchestra.run.smithi143:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-14T05:32:13.084 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/vg_nvme/lv_4 2024-09-14T05:32:13.140 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-14T05:32:13.140 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-14T05:32:13.140 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-14T05:32:13.140 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-14T05:32:13.140 INFO:teuthology.orchestra.run.smithi143.stdout:Context: system_u:object_r:device_t:s0 2024-09-14T05:32:13.140 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2024-09-14 05:31:46.224929686 +0000 2024-09-14T05:32:13.141 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2024-09-14 05:29:17.632138604 +0000 2024-09-14T05:32:13.141 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2024-09-14 05:29:17.632138604 +0000 2024-09-14T05:32:13.141 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: 2024-09-14 05:29:17.632138604 +0000 2024-09-14T05:32:13.141 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-14T05:32:13.206 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2024-09-14T05:32:13.206 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2024-09-14T05:32:13.206 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.000280815 s, 1.8 MB/s 2024-09-14T05:32:13.207 DEBUG:teuthology.orchestra.run.smithi143:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-14T05:32:13.264 INFO:tasks.ceph:osd dev map: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-14T05:32:13.264 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi143.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-14T05:32:13.264 INFO:tasks.ceph:Generating config... 2024-09-14T05:32:13.266 INFO:tasks.ceph:[client] debug rbd = 20 2024-09-14T05:32:13.266 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-09-14T05:32:13.266 INFO:tasks.ceph:[client] log to stderr = False 2024-09-14T05:32:13.266 INFO:tasks.ceph:[client] rbd default features = 125 2024-09-14T05:32:13.266 INFO:tasks.ceph:[client] rbd mirror image policy type = none 2024-09-14T05:32:13.266 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T05:32:13.266 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T05:32:13.266 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T05:32:13.266 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T05:32:13.266 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T05:32:13.267 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T05:32:13.267 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T05:32:13.267 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T05:32:13.267 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-14T05:32:13.267 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-09-14T05:32:13.267 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-14T05:32:13.267 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-14T05:32:13.267 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-14T05:32:13.267 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-14T05:32:13.267 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-14T05:32:13.267 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-14T05:32:13.267 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-14T05:32:13.268 INFO:tasks.ceph:[osd] bluestore compression algorithm = zstd 2024-09-14T05:32:13.268 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-14T05:32:13.268 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-14T05:32:13.268 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-14T05:32:13.268 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-14T05:32:13.268 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-14T05:32:13.268 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-14T05:32:13.268 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-14T05:32:13.268 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-14T05:32:13.268 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-14T05:32:13.268 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-14T05:32:13.268 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-14T05:32:13.269 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-14T05:32:13.269 INFO:tasks.ceph:Setting up cluster1.mon.a... 2024-09-14T05:32:13.269 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster1.keyring 2024-09-14T05:32:13.351 INFO:teuthology.orchestra.run.smithi143.stdout:creating /etc/ceph/cluster1.keyring 2024-09-14T05:32:13.353 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster1.keyring 2024-09-14T05:32:13.436 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-09-14T05:32:13.501 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster1.mon.a', '172.21.15.143')] 2024-09-14T05:32:13.502 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.143', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-09-14T05:32:13.502 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-14T05:32:13.503 DEBUG:teuthology.orchestra.run.smithi143:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-14T05:32:13.558 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.143 --print /home/ubuntu/cephtest/cluster1.monmap 2024-09-14T05:32:13.637 INFO:teuthology.orchestra.run.smithi143.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster1.monmap 2024-09-14T05:32:13.638 INFO:teuthology.orchestra.run.smithi143.stdout:monmaptool: generated fsid 5628b0ee-4376-46aa-afba-0841d836b50f 2024-09-14T05:32:13.638 INFO:teuthology.orchestra.run.smithi143.stdout:setting min_mon_release = quincy 2024-09-14T05:32:13.638 INFO:teuthology.orchestra.run.smithi143.stdout:epoch 0 2024-09-14T05:32:13.638 INFO:teuthology.orchestra.run.smithi143.stdout:fsid 5628b0ee-4376-46aa-afba-0841d836b50f 2024-09-14T05:32:13.638 INFO:teuthology.orchestra.run.smithi143.stdout:last_changed 2024-09-14T05:32:13.637628+0000 2024-09-14T05:32:13.638 INFO:teuthology.orchestra.run.smithi143.stdout:created 2024-09-14T05:32:13.637628+0000 2024-09-14T05:32:13.638 INFO:teuthology.orchestra.run.smithi143.stdout:min_mon_release 17 (quincy) 2024-09-14T05:32:13.638 INFO:teuthology.orchestra.run.smithi143.stdout:election_strategy: 1 2024-09-14T05:32:13.638 INFO:teuthology.orchestra.run.smithi143.stdout:0: [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon.a 2024-09-14T05:32:13.638 INFO:teuthology.orchestra.run.smithi143.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster1.monmap (1 monitors) 2024-09-14T05:32:13.640 DEBUG:teuthology.orchestra.run.smithi143:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-14T05:32:13.695 INFO:tasks.ceph:Writing /etc/ceph/cluster1.conf for FSID 5628b0ee-4376-46aa-afba-0841d836b50f... 2024-09-14T05:32:13.696 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-09-14T05:32:13.738 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-09-14T05:32:13.777 INFO:teuthology.orchestra.run.smithi143.stdout:[global] 2024-09-14T05:32:13.778 INFO:teuthology.orchestra.run.smithi143.stdout: chdir = "" 2024-09-14T05:32:13.778 INFO:teuthology.orchestra.run.smithi143.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-14T05:32:13.778 INFO:teuthology.orchestra.run.smithi143.stdout: auth supported = cephx 2024-09-14T05:32:13.778 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:13.778 INFO:teuthology.orchestra.run.smithi143.stdout: filestore xattr use omap = true 2024-09-14T05:32:13.778 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:13.778 INFO:teuthology.orchestra.run.smithi143.stdout: mon clock drift allowed = 1.000 2024-09-14T05:32:13.778 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:13.779 INFO:teuthology.orchestra.run.smithi143.stdout: osd crush chooseleaf type = 0 2024-09-14T05:32:13.779 INFO:teuthology.orchestra.run.smithi143.stdout: auth debug = true 2024-09-14T05:32:13.779 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:13.779 INFO:teuthology.orchestra.run.smithi143.stdout: ms die on old message = true 2024-09-14T05:32:13.779 INFO:teuthology.orchestra.run.smithi143.stdout: ms die on bug = true 2024-09-14T05:32:13.779 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:13.779 INFO:teuthology.orchestra.run.smithi143.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-14T05:32:13.779 INFO:teuthology.orchestra.run.smithi143.stdout: mon pg warn max object skew = 0 2024-09-14T05:32:13.780 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:13.780 INFO:teuthology.orchestra.run.smithi143.stdout: # disable pg_autoscaler by default for new pools 2024-09-14T05:32:13.780 INFO:teuthology.orchestra.run.smithi143.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-14T05:32:13.780 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:13.780 INFO:teuthology.orchestra.run.smithi143.stdout: osd pool default size = 2 2024-09-14T05:32:13.780 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:13.780 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd allow primary affinity = true 2024-09-14T05:32:13.780 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd allow pg remap = true 2024-09-14T05:32:13.780 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on legacy crush tunables = false 2024-09-14T05:32:13.780 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on crush straw calc version zero = false 2024-09-14T05:32:13.780 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on no sortbitwise = false 2024-09-14T05:32:13.780 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on osd down out interval zero = false 2024-09-14T05:32:13.781 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on too few osds = false 2024-09-14T05:32:13.781 INFO:teuthology.orchestra.run.smithi143.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-14T05:32:13.781 INFO:teuthology.orchestra.run.smithi143.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-14T05:32:13.781 INFO:teuthology.orchestra.run.smithi143.stdout: mon_allow_pool_size_one = true 2024-09-14T05:32:13.781 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:13.781 INFO:teuthology.orchestra.run.smithi143.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-14T05:32:13.781 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:13.781 INFO:teuthology.orchestra.run.smithi143.stdout: osd default data pool replay window = 5 2024-09-14T05:32:13.781 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:13.781 INFO:teuthology.orchestra.run.smithi143.stdout: mon allow pool delete = true 2024-09-14T05:32:13.781 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:13.781 INFO:teuthology.orchestra.run.smithi143.stdout: mon cluster log file level = debug 2024-09-14T05:32:13.781 INFO:teuthology.orchestra.run.smithi143.stdout: debug asserts on shutdown = true 2024-09-14T05:32:13.782 INFO:teuthology.orchestra.run.smithi143.stdout: mon health detail to clog = false 2024-09-14T05:32:13.782 INFO:teuthology.orchestra.run.smithi143.stdout: mon host = 172.21.15.143 2024-09-14T05:32:13.782 INFO:teuthology.orchestra.run.smithi143.stdout: mon client directed command retry = 5 2024-09-14T05:32:13.782 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on pool no app = False 2024-09-14T05:32:13.782 INFO:teuthology.orchestra.run.smithi143.stdout: ms inject socket failures = 5000 2024-09-14T05:32:13.782 INFO:teuthology.orchestra.run.smithi143.stdout: fsid = 5628b0ee-4376-46aa-afba-0841d836b50f 2024-09-14T05:32:13.782 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:13.782 INFO:teuthology.orchestra.run.smithi143.stdout:[osd] 2024-09-14T05:32:13.782 INFO:teuthology.orchestra.run.smithi143.stdout: osd journal size = 100 2024-09-14T05:32:13.782 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:13.782 INFO:teuthology.orchestra.run.smithi143.stdout: osd scrub load threshold = 5.0 2024-09-14T05:32:13.782 INFO:teuthology.orchestra.run.smithi143.stdout: osd scrub max interval = 600 2024-09-14T05:32:13.782 INFO:teuthology.orchestra.run.smithi143.stdout: osd mclock profile = high_recovery_ops 2024-09-14T05:32:13.782 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:13.783 INFO:teuthology.orchestra.run.smithi143.stdout: osd recover clone overlap = true 2024-09-14T05:32:13.783 INFO:teuthology.orchestra.run.smithi143.stdout: osd recovery max chunk = 1048576 2024-09-14T05:32:13.783 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:13.783 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug shutdown = true 2024-09-14T05:32:13.783 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug op order = true 2024-09-14T05:32:13.783 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug verify stray on activate = true 2024-09-14T05:32:13.783 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug trim objects = true 2024-09-14T05:32:13.783 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:13.783 INFO:teuthology.orchestra.run.smithi143.stdout: osd open classes on start = true 2024-09-14T05:32:13.783 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug pg log writeout = true 2024-09-14T05:32:13.783 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:13.783 INFO:teuthology.orchestra.run.smithi143.stdout: osd deep scrub update digest min age = 30 2024-09-14T05:32:13.783 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:13.784 INFO:teuthology.orchestra.run.smithi143.stdout: osd map max advance = 10 2024-09-14T05:32:13.784 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:13.784 INFO:teuthology.orchestra.run.smithi143.stdout: journal zero on create = true 2024-09-14T05:32:13.784 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:13.784 INFO:teuthology.orchestra.run.smithi143.stdout: filestore ondisk finisher threads = 3 2024-09-14T05:32:13.784 INFO:teuthology.orchestra.run.smithi143.stdout: filestore apply finisher threads = 3 2024-09-14T05:32:13.784 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:13.784 INFO:teuthology.orchestra.run.smithi143.stdout: bdev debug aio = true 2024-09-14T05:32:13.784 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug misdirected ops = true 2024-09-14T05:32:13.784 INFO:teuthology.orchestra.run.smithi143.stdout: bluestore block size = 96636764160 2024-09-14T05:32:13.784 INFO:teuthology.orchestra.run.smithi143.stdout: bluestore compression algorithm = zstd 2024-09-14T05:32:13.784 INFO:teuthology.orchestra.run.smithi143.stdout: bluestore compression mode = aggressive 2024-09-14T05:32:13.784 INFO:teuthology.orchestra.run.smithi143.stdout: bluestore fsck on mount = True 2024-09-14T05:32:13.785 INFO:teuthology.orchestra.run.smithi143.stdout: debug bluefs = 1/20 2024-09-14T05:32:13.785 INFO:teuthology.orchestra.run.smithi143.stdout: debug bluestore = 1/20 2024-09-14T05:32:13.785 INFO:teuthology.orchestra.run.smithi143.stdout: debug ms = 1 2024-09-14T05:32:13.785 INFO:teuthology.orchestra.run.smithi143.stdout: debug osd = 20 2024-09-14T05:32:13.785 INFO:teuthology.orchestra.run.smithi143.stdout: debug rocksdb = 4/10 2024-09-14T05:32:13.785 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd backfillfull_ratio = 0.85 2024-09-14T05:32:13.786 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd full ratio = 0.9 2024-09-14T05:32:13.786 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd nearfull ratio = 0.8 2024-09-14T05:32:13.786 INFO:teuthology.orchestra.run.smithi143.stdout: osd failsafe full ratio = 0.95 2024-09-14T05:32:13.786 INFO:teuthology.orchestra.run.smithi143.stdout: osd objectstore = bluestore 2024-09-14T05:32:13.786 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:13.786 INFO:teuthology.orchestra.run.smithi143.stdout:[mgr] 2024-09-14T05:32:13.786 INFO:teuthology.orchestra.run.smithi143.stdout: debug ms = 1 2024-09-14T05:32:13.786 INFO:teuthology.orchestra.run.smithi143.stdout: debug mgr = 20 2024-09-14T05:32:13.786 INFO:teuthology.orchestra.run.smithi143.stdout: debug mon = 20 2024-09-14T05:32:13.786 INFO:teuthology.orchestra.run.smithi143.stdout: debug auth = 20 2024-09-14T05:32:13.786 INFO:teuthology.orchestra.run.smithi143.stdout: mon reweight min pgs per osd = 4 2024-09-14T05:32:13.787 INFO:teuthology.orchestra.run.smithi143.stdout: mon reweight min bytes per osd = 10 2024-09-14T05:32:13.787 INFO:teuthology.orchestra.run.smithi143.stdout: mgr/telemetry/nag = false 2024-09-14T05:32:13.787 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:13.787 INFO:teuthology.orchestra.run.smithi143.stdout:[mon] 2024-09-14T05:32:13.787 INFO:teuthology.orchestra.run.smithi143.stdout: debug ms = 1 2024-09-14T05:32:13.787 INFO:teuthology.orchestra.run.smithi143.stdout: debug mon = 20 2024-09-14T05:32:13.787 INFO:teuthology.orchestra.run.smithi143.stdout: debug paxos = 20 2024-09-14T05:32:13.787 INFO:teuthology.orchestra.run.smithi143.stdout: debug auth = 20 2024-09-14T05:32:13.787 INFO:teuthology.orchestra.run.smithi143.stdout: mon data avail warn = 5 2024-09-14T05:32:13.788 INFO:teuthology.orchestra.run.smithi143.stdout: mon mgr mkfs grace = 240 2024-09-14T05:32:13.788 INFO:teuthology.orchestra.run.smithi143.stdout: mon reweight min pgs per osd = 4 2024-09-14T05:32:13.788 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd reporter subtree level = osd 2024-09-14T05:32:13.788 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd prime pg temp = true 2024-09-14T05:32:13.788 INFO:teuthology.orchestra.run.smithi143.stdout: mon reweight min bytes per osd = 10 2024-09-14T05:32:13.788 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:13.788 INFO:teuthology.orchestra.run.smithi143.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-14T05:32:13.789 INFO:teuthology.orchestra.run.smithi143.stdout: auth mon ticket ttl = 660 # 11m 2024-09-14T05:32:13.789 INFO:teuthology.orchestra.run.smithi143.stdout: auth service ticket ttl = 240 # 4m 2024-09-14T05:32:13.789 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:13.789 INFO:teuthology.orchestra.run.smithi143.stdout: # don't complain about insecure global_id in the test suite 2024-09-14T05:32:13.789 INFO:teuthology.orchestra.run.smithi143.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-14T05:32:13.789 INFO:teuthology.orchestra.run.smithi143.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-14T05:32:13.789 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:13.789 INFO:teuthology.orchestra.run.smithi143.stdout: # 1m isn't quite enough 2024-09-14T05:32:13.789 INFO:teuthology.orchestra.run.smithi143.stdout: mon_down_mkfs_grace = 2m 2024-09-14T05:32:13.789 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:13.790 INFO:teuthology.orchestra.run.smithi143.stdout: mon_warn_on_filestore_osds = false 2024-09-14T05:32:13.790 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:13.790 INFO:teuthology.orchestra.run.smithi143.stdout:[client] 2024-09-14T05:32:13.790 INFO:teuthology.orchestra.run.smithi143.stdout: rgw cache enabled = true 2024-09-14T05:32:13.790 INFO:teuthology.orchestra.run.smithi143.stdout: rgw enable ops log = true 2024-09-14T05:32:13.790 INFO:teuthology.orchestra.run.smithi143.stdout: rgw enable usage log = true 2024-09-14T05:32:13.790 INFO:teuthology.orchestra.run.smithi143.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-14T05:32:13.790 INFO:teuthology.orchestra.run.smithi143.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-14T05:32:13.790 INFO:teuthology.orchestra.run.smithi143.stdout: debug rbd = 20 2024-09-14T05:32:13.790 INFO:teuthology.orchestra.run.smithi143.stdout: debug rbd_mirror = 15 2024-09-14T05:32:13.790 INFO:teuthology.orchestra.run.smithi143.stdout: log to stderr = False 2024-09-14T05:32:13.790 INFO:teuthology.orchestra.run.smithi143.stdout: rbd default features = 125 2024-09-14T05:32:13.790 INFO:teuthology.orchestra.run.smithi143.stdout: rbd mirror image policy type = none 2024-09-14T05:32:13.790 INFO:teuthology.orchestra.run.smithi143.stdout:[mon.a] 2024-09-14T05:32:13.791 INFO:teuthology.orchestra.run.smithi143.stdout:[client.mirror.0] 2024-09-14T05:32:13.791 INFO:teuthology.orchestra.run.smithi143.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T05:32:13.791 INFO:teuthology.orchestra.run.smithi143.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T05:32:13.791 INFO:teuthology.orchestra.run.smithi143.stdout:[client.mirror.1] 2024-09-14T05:32:13.791 INFO:teuthology.orchestra.run.smithi143.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T05:32:13.791 INFO:teuthology.orchestra.run.smithi143.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T05:32:13.791 INFO:teuthology.orchestra.run.smithi143.stdout:[client.mirror.2] 2024-09-14T05:32:13.791 INFO:teuthology.orchestra.run.smithi143.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T05:32:13.791 INFO:teuthology.orchestra.run.smithi143.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T05:32:13.791 INFO:teuthology.orchestra.run.smithi143.stdout:[client.mirror.3] 2024-09-14T05:32:13.791 INFO:teuthology.orchestra.run.smithi143.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T05:32:13.791 INFO:teuthology.orchestra.run.smithi143.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T05:32:13.792 INFO:teuthology.orchestra.run.smithi191.stdout:[global] 2024-09-14T05:32:13.792 INFO:teuthology.orchestra.run.smithi191.stdout: chdir = "" 2024-09-14T05:32:13.792 INFO:teuthology.orchestra.run.smithi191.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-14T05:32:13.793 INFO:teuthology.orchestra.run.smithi191.stdout: auth supported = cephx 2024-09-14T05:32:13.793 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:13.793 INFO:teuthology.orchestra.run.smithi191.stdout: filestore xattr use omap = true 2024-09-14T05:32:13.793 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:13.793 INFO:teuthology.orchestra.run.smithi191.stdout: mon clock drift allowed = 1.000 2024-09-14T05:32:13.793 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:13.793 INFO:teuthology.orchestra.run.smithi191.stdout: osd crush chooseleaf type = 0 2024-09-14T05:32:13.793 INFO:teuthology.orchestra.run.smithi191.stdout: auth debug = true 2024-09-14T05:32:13.793 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:13.793 INFO:teuthology.orchestra.run.smithi191.stdout: ms die on old message = true 2024-09-14T05:32:13.793 INFO:teuthology.orchestra.run.smithi191.stdout: ms die on bug = true 2024-09-14T05:32:13.793 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:13.794 INFO:teuthology.orchestra.run.smithi191.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-14T05:32:13.794 INFO:teuthology.orchestra.run.smithi191.stdout: mon pg warn max object skew = 0 2024-09-14T05:32:13.794 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:13.794 INFO:teuthology.orchestra.run.smithi191.stdout: # disable pg_autoscaler by default for new pools 2024-09-14T05:32:13.794 INFO:teuthology.orchestra.run.smithi191.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-14T05:32:13.794 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:13.794 INFO:teuthology.orchestra.run.smithi191.stdout: osd pool default size = 2 2024-09-14T05:32:13.794 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:13.794 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd allow primary affinity = true 2024-09-14T05:32:13.794 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd allow pg remap = true 2024-09-14T05:32:13.794 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on legacy crush tunables = false 2024-09-14T05:32:13.795 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on crush straw calc version zero = false 2024-09-14T05:32:13.795 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on no sortbitwise = false 2024-09-14T05:32:13.795 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on osd down out interval zero = false 2024-09-14T05:32:13.795 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on too few osds = false 2024-09-14T05:32:13.795 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-14T05:32:13.795 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-14T05:32:13.795 INFO:teuthology.orchestra.run.smithi191.stdout: mon_allow_pool_size_one = true 2024-09-14T05:32:13.796 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:13.796 INFO:teuthology.orchestra.run.smithi191.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-14T05:32:13.796 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:13.796 INFO:teuthology.orchestra.run.smithi191.stdout: osd default data pool replay window = 5 2024-09-14T05:32:13.796 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:13.796 INFO:teuthology.orchestra.run.smithi191.stdout: mon allow pool delete = true 2024-09-14T05:32:13.796 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:13.796 INFO:teuthology.orchestra.run.smithi191.stdout: mon cluster log file level = debug 2024-09-14T05:32:13.796 INFO:teuthology.orchestra.run.smithi191.stdout: debug asserts on shutdown = true 2024-09-14T05:32:13.796 INFO:teuthology.orchestra.run.smithi191.stdout: mon health detail to clog = false 2024-09-14T05:32:13.796 INFO:teuthology.orchestra.run.smithi191.stdout: mon host = 172.21.15.143 2024-09-14T05:32:13.796 INFO:teuthology.orchestra.run.smithi191.stdout: mon client directed command retry = 5 2024-09-14T05:32:13.797 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on pool no app = False 2024-09-14T05:32:13.797 INFO:teuthology.orchestra.run.smithi191.stdout: ms inject socket failures = 5000 2024-09-14T05:32:13.797 INFO:teuthology.orchestra.run.smithi191.stdout: fsid = 5628b0ee-4376-46aa-afba-0841d836b50f 2024-09-14T05:32:13.797 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:13.797 INFO:teuthology.orchestra.run.smithi191.stdout:[osd] 2024-09-14T05:32:13.797 INFO:teuthology.orchestra.run.smithi191.stdout: osd journal size = 100 2024-09-14T05:32:13.797 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:13.797 INFO:teuthology.orchestra.run.smithi191.stdout: osd scrub load threshold = 5.0 2024-09-14T05:32:13.797 INFO:teuthology.orchestra.run.smithi191.stdout: osd scrub max interval = 600 2024-09-14T05:32:13.797 INFO:teuthology.orchestra.run.smithi191.stdout: osd mclock profile = high_recovery_ops 2024-09-14T05:32:13.798 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:13.798 INFO:teuthology.orchestra.run.smithi191.stdout: osd recover clone overlap = true 2024-09-14T05:32:13.798 INFO:teuthology.orchestra.run.smithi191.stdout: osd recovery max chunk = 1048576 2024-09-14T05:32:13.798 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:13.798 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug shutdown = true 2024-09-14T05:32:13.798 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug op order = true 2024-09-14T05:32:13.798 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug verify stray on activate = true 2024-09-14T05:32:13.798 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug trim objects = true 2024-09-14T05:32:13.798 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:13.798 INFO:teuthology.orchestra.run.smithi191.stdout: osd open classes on start = true 2024-09-14T05:32:13.798 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug pg log writeout = true 2024-09-14T05:32:13.798 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:13.798 INFO:teuthology.orchestra.run.smithi191.stdout: osd deep scrub update digest min age = 30 2024-09-14T05:32:13.798 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:13.799 INFO:teuthology.orchestra.run.smithi191.stdout: osd map max advance = 10 2024-09-14T05:32:13.799 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:13.799 INFO:teuthology.orchestra.run.smithi191.stdout: journal zero on create = true 2024-09-14T05:32:13.799 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:13.799 INFO:teuthology.orchestra.run.smithi191.stdout: filestore ondisk finisher threads = 3 2024-09-14T05:32:13.799 INFO:teuthology.orchestra.run.smithi191.stdout: filestore apply finisher threads = 3 2024-09-14T05:32:13.799 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:13.799 INFO:teuthology.orchestra.run.smithi191.stdout: bdev debug aio = true 2024-09-14T05:32:13.799 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug misdirected ops = true 2024-09-14T05:32:13.799 INFO:teuthology.orchestra.run.smithi191.stdout: bluestore block size = 96636764160 2024-09-14T05:32:13.799 INFO:teuthology.orchestra.run.smithi191.stdout: bluestore compression algorithm = zstd 2024-09-14T05:32:13.799 INFO:teuthology.orchestra.run.smithi191.stdout: bluestore compression mode = aggressive 2024-09-14T05:32:13.799 INFO:teuthology.orchestra.run.smithi191.stdout: bluestore fsck on mount = True 2024-09-14T05:32:13.799 INFO:teuthology.orchestra.run.smithi191.stdout: debug bluefs = 1/20 2024-09-14T05:32:13.800 INFO:teuthology.orchestra.run.smithi191.stdout: debug bluestore = 1/20 2024-09-14T05:32:13.800 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2024-09-14T05:32:13.800 INFO:teuthology.orchestra.run.smithi191.stdout: debug osd = 20 2024-09-14T05:32:13.800 INFO:teuthology.orchestra.run.smithi191.stdout: debug rocksdb = 4/10 2024-09-14T05:32:13.800 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd backfillfull_ratio = 0.85 2024-09-14T05:32:13.800 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd full ratio = 0.9 2024-09-14T05:32:13.800 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd nearfull ratio = 0.8 2024-09-14T05:32:13.800 INFO:teuthology.orchestra.run.smithi191.stdout: osd failsafe full ratio = 0.95 2024-09-14T05:32:13.800 INFO:teuthology.orchestra.run.smithi191.stdout: osd objectstore = bluestore 2024-09-14T05:32:13.800 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:13.800 INFO:teuthology.orchestra.run.smithi191.stdout:[mgr] 2024-09-14T05:32:13.800 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2024-09-14T05:32:13.800 INFO:teuthology.orchestra.run.smithi191.stdout: debug mgr = 20 2024-09-14T05:32:13.801 INFO:teuthology.orchestra.run.smithi191.stdout: debug mon = 20 2024-09-14T05:32:13.801 INFO:teuthology.orchestra.run.smithi191.stdout: debug auth = 20 2024-09-14T05:32:13.801 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min pgs per osd = 4 2024-09-14T05:32:13.801 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min bytes per osd = 10 2024-09-14T05:32:13.801 INFO:teuthology.orchestra.run.smithi191.stdout: mgr/telemetry/nag = false 2024-09-14T05:32:13.801 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:13.801 INFO:teuthology.orchestra.run.smithi191.stdout:[mon] 2024-09-14T05:32:13.801 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2024-09-14T05:32:13.801 INFO:teuthology.orchestra.run.smithi191.stdout: debug mon = 20 2024-09-14T05:32:13.801 INFO:teuthology.orchestra.run.smithi191.stdout: debug paxos = 20 2024-09-14T05:32:13.801 INFO:teuthology.orchestra.run.smithi191.stdout: debug auth = 20 2024-09-14T05:32:13.801 INFO:teuthology.orchestra.run.smithi191.stdout: mon data avail warn = 5 2024-09-14T05:32:13.802 INFO:teuthology.orchestra.run.smithi191.stdout: mon mgr mkfs grace = 240 2024-09-14T05:32:13.802 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min pgs per osd = 4 2024-09-14T05:32:13.802 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd reporter subtree level = osd 2024-09-14T05:32:13.802 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd prime pg temp = true 2024-09-14T05:32:13.802 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min bytes per osd = 10 2024-09-14T05:32:13.802 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:13.802 INFO:teuthology.orchestra.run.smithi191.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-14T05:32:13.802 INFO:teuthology.orchestra.run.smithi191.stdout: auth mon ticket ttl = 660 # 11m 2024-09-14T05:32:13.802 INFO:teuthology.orchestra.run.smithi191.stdout: auth service ticket ttl = 240 # 4m 2024-09-14T05:32:13.802 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:13.802 INFO:teuthology.orchestra.run.smithi191.stdout: # don't complain about insecure global_id in the test suite 2024-09-14T05:32:13.802 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-14T05:32:13.802 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-14T05:32:13.802 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:13.803 INFO:teuthology.orchestra.run.smithi191.stdout: # 1m isn't quite enough 2024-09-14T05:32:13.803 INFO:teuthology.orchestra.run.smithi191.stdout: mon_down_mkfs_grace = 2m 2024-09-14T05:32:13.803 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:13.803 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_filestore_osds = false 2024-09-14T05:32:13.803 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:13.803 INFO:teuthology.orchestra.run.smithi191.stdout:[client] 2024-09-14T05:32:13.803 INFO:teuthology.orchestra.run.smithi191.stdout: rgw cache enabled = true 2024-09-14T05:32:13.803 INFO:teuthology.orchestra.run.smithi191.stdout: rgw enable ops log = true 2024-09-14T05:32:13.803 INFO:teuthology.orchestra.run.smithi191.stdout: rgw enable usage log = true 2024-09-14T05:32:13.803 INFO:teuthology.orchestra.run.smithi191.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-14T05:32:13.803 INFO:teuthology.orchestra.run.smithi191.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-14T05:32:13.803 INFO:teuthology.orchestra.run.smithi191.stdout: debug rbd = 20 2024-09-14T05:32:13.803 INFO:teuthology.orchestra.run.smithi191.stdout: debug rbd_mirror = 15 2024-09-14T05:32:13.803 INFO:teuthology.orchestra.run.smithi191.stdout: log to stderr = False 2024-09-14T05:32:13.804 INFO:teuthology.orchestra.run.smithi191.stdout: rbd default features = 125 2024-09-14T05:32:13.804 INFO:teuthology.orchestra.run.smithi191.stdout: rbd mirror image policy type = none 2024-09-14T05:32:13.804 INFO:teuthology.orchestra.run.smithi191.stdout:[mon.a] 2024-09-14T05:32:13.804 INFO:teuthology.orchestra.run.smithi191.stdout:[client.mirror.0] 2024-09-14T05:32:13.804 INFO:teuthology.orchestra.run.smithi191.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T05:32:13.804 INFO:teuthology.orchestra.run.smithi191.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T05:32:13.804 INFO:teuthology.orchestra.run.smithi191.stdout:[client.mirror.1] 2024-09-14T05:32:13.804 INFO:teuthology.orchestra.run.smithi191.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T05:32:13.804 INFO:teuthology.orchestra.run.smithi191.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T05:32:13.804 INFO:teuthology.orchestra.run.smithi191.stdout:[client.mirror.2] 2024-09-14T05:32:13.804 INFO:teuthology.orchestra.run.smithi191.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T05:32:13.804 INFO:teuthology.orchestra.run.smithi191.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T05:32:13.804 INFO:teuthology.orchestra.run.smithi191.stdout:[client.mirror.3] 2024-09-14T05:32:13.805 INFO:teuthology.orchestra.run.smithi191.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T05:32:13.805 INFO:teuthology.orchestra.run.smithi191.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T05:32:13.806 INFO:tasks.ceph:Creating admin key on cluster1.mon.a... 2024-09-14T05:32:13.806 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster1.keyring 2024-09-14T05:32:13.901 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-14T05:32:13.902 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-14T05:32:13.902 DEBUG:teuthology.orchestra.run.smithi143:> dd if=/etc/ceph/cluster1.keyring of=/dev/stdout 2024-09-14T05:32:13.917 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-14T05:32:13.917 DEBUG:teuthology.orchestra.run.smithi143:> dd if=/home/ubuntu/cephtest/cluster1.monmap of=/dev/stdout 2024-09-14T05:32:13.974 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi143.front.sepia.ceph.com 2024-09-14T05:32:13.974 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-14T05:32:13.974 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/etc/ceph/cluster1.keyring 2024-09-14T05:32:13.974 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-09-14T05:32:14.049 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-14T05:32:14.050 DEBUG:teuthology.orchestra.run.smithi143:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-09-14T05:32:14.106 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi191.front.sepia.ceph.com 2024-09-14T05:32:14.106 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-14T05:32:14.106 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/ceph/cluster1.keyring 2024-09-14T05:32:14.106 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-09-14T05:32:14.147 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-14T05:32:14.147 DEBUG:teuthology.orchestra.run.smithi191:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-09-14T05:32:14.206 INFO:tasks.ceph:Setting up mon nodes... 2024-09-14T05:32:14.206 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-14T05:32:14.207 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /var/lib/ceph/mgr/cluster1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster1-x/keyring 2024-09-14T05:32:14.257 INFO:teuthology.orchestra.run.smithi143.stdout:creating /var/lib/ceph/mgr/cluster1-x/keyring 2024-09-14T05:32:14.259 INFO:tasks.ceph:Setting up mds nodes... 2024-09-14T05:32:14.260 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-14T05:32:14.260 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster1.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.0.keyring 2024-09-14T05:32:14.300 INFO:teuthology.orchestra.run.smithi143.stdout:creating /etc/ceph/cluster1.client.0.keyring 2024-09-14T05:32:14.313 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster1.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.keyring 2024-09-14T05:32:14.376 INFO:teuthology.orchestra.run.smithi191.stdout:creating /etc/ceph/cluster1.client.mirror.keyring 2024-09-14T05:32:14.390 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster1.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.0.keyring 2024-09-14T05:32:14.433 INFO:teuthology.orchestra.run.smithi191.stdout:creating /etc/ceph/cluster1.client.mirror.0.keyring 2024-09-14T05:32:14.446 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster1.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.1.keyring 2024-09-14T05:32:14.529 INFO:teuthology.orchestra.run.smithi191.stdout:creating /etc/ceph/cluster1.client.mirror.1.keyring 2024-09-14T05:32:14.542 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster1.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.2.keyring 2024-09-14T05:32:14.624 INFO:teuthology.orchestra.run.smithi191.stdout:creating /etc/ceph/cluster1.client.mirror.2.keyring 2024-09-14T05:32:14.637 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster1.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.3.keyring 2024-09-14T05:32:14.718 INFO:teuthology.orchestra.run.smithi191.stdout:creating /etc/ceph/cluster1.client.mirror.3.keyring 2024-09-14T05:32:14.731 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster1.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.4.keyring 2024-09-14T05:32:14.811 INFO:teuthology.orchestra.run.smithi191.stdout:creating /etc/ceph/cluster1.client.mirror.4.keyring 2024-09-14T05:32:14.824 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster1.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.5.keyring 2024-09-14T05:32:14.906 INFO:teuthology.orchestra.run.smithi191.stdout:creating /etc/ceph/cluster1.client.mirror.5.keyring 2024-09-14T05:32:14.918 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster1.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.6.keyring 2024-09-14T05:32:15.000 INFO:teuthology.orchestra.run.smithi191.stdout:creating /etc/ceph/cluster1.client.mirror.6.keyring 2024-09-14T05:32:15.013 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-14T05:32:15.013 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi143.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-14T05:32:15.013 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /var/lib/ceph/osd/cluster1-0 2024-09-14T05:32:15.040 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-14T05:32:15.040 INFO:tasks.ceph:role: cluster1.osd.0 2024-09-14T05:32:15.040 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi143.front.sepia.ceph.com 2024-09-14T05:32:15.040 DEBUG:teuthology.orchestra.run.smithi143:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-14T05:32:15.115 INFO:teuthology.orchestra.run.smithi143.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-14T05:32:15.115 INFO:teuthology.orchestra.run.smithi143.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-14T05:32:15.115 INFO:teuthology.orchestra.run.smithi143.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-14T05:32:15.115 INFO:teuthology.orchestra.run.smithi143.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-14T05:32:15.115 INFO:teuthology.orchestra.run.smithi143.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-14T05:32:15.115 INFO:teuthology.orchestra.run.smithi143.stdout: = sunit=0 swidth=0 blks 2024-09-14T05:32:15.115 INFO:teuthology.orchestra.run.smithi143.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-14T05:32:15.115 INFO:teuthology.orchestra.run.smithi143.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-14T05:32:15.115 INFO:teuthology.orchestra.run.smithi143.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-14T05:32:15.115 INFO:teuthology.orchestra.run.smithi143.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-14T05:32:15.611 INFO:teuthology.orchestra.run.smithi143.stdout:Discarding blocks...Done. 2024-09-14T05:32:15.614 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi143.front.sepia.ceph.com -o noatime 2024-09-14T05:32:15.614 DEBUG:teuthology.orchestra.run.smithi143:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster1-0 2024-09-14T05:32:15.759 DEBUG:teuthology.orchestra.run.smithi143:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-0 2024-09-14T05:32:15.792 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /var/lib/ceph/osd/cluster1-1 2024-09-14T05:32:15.859 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-14T05:32:15.859 INFO:tasks.ceph:role: cluster1.osd.1 2024-09-14T05:32:15.859 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi143.front.sepia.ceph.com 2024-09-14T05:32:15.859 DEBUG:teuthology.orchestra.run.smithi143:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-14T05:32:15.933 INFO:teuthology.orchestra.run.smithi143.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-14T05:32:15.933 INFO:teuthology.orchestra.run.smithi143.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-14T05:32:15.934 INFO:teuthology.orchestra.run.smithi143.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-14T05:32:15.934 INFO:teuthology.orchestra.run.smithi143.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-14T05:32:15.934 INFO:teuthology.orchestra.run.smithi143.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-14T05:32:15.934 INFO:teuthology.orchestra.run.smithi143.stdout: = sunit=0 swidth=0 blks 2024-09-14T05:32:15.934 INFO:teuthology.orchestra.run.smithi143.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-14T05:32:15.934 INFO:teuthology.orchestra.run.smithi143.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-14T05:32:15.934 INFO:teuthology.orchestra.run.smithi143.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-14T05:32:15.934 INFO:teuthology.orchestra.run.smithi143.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-14T05:32:16.384 INFO:teuthology.orchestra.run.smithi143.stdout:Discarding blocks...Done. 2024-09-14T05:32:16.386 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi143.front.sepia.ceph.com -o noatime 2024-09-14T05:32:16.386 DEBUG:teuthology.orchestra.run.smithi143:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster1-1 2024-09-14T05:32:16.471 DEBUG:teuthology.orchestra.run.smithi143:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-1 2024-09-14T05:32:16.542 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /var/lib/ceph/osd/cluster1-2 2024-09-14T05:32:16.608 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-14T05:32:16.608 INFO:tasks.ceph:role: cluster1.osd.2 2024-09-14T05:32:16.608 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi143.front.sepia.ceph.com 2024-09-14T05:32:16.608 DEBUG:teuthology.orchestra.run.smithi143:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-14T05:32:16.682 INFO:teuthology.orchestra.run.smithi143.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-14T05:32:16.682 INFO:teuthology.orchestra.run.smithi143.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-14T05:32:16.682 INFO:teuthology.orchestra.run.smithi143.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-14T05:32:16.682 INFO:teuthology.orchestra.run.smithi143.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-14T05:32:16.682 INFO:teuthology.orchestra.run.smithi143.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-14T05:32:16.682 INFO:teuthology.orchestra.run.smithi143.stdout: = sunit=0 swidth=0 blks 2024-09-14T05:32:16.682 INFO:teuthology.orchestra.run.smithi143.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-14T05:32:16.682 INFO:teuthology.orchestra.run.smithi143.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-14T05:32:16.682 INFO:teuthology.orchestra.run.smithi143.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-14T05:32:16.683 INFO:teuthology.orchestra.run.smithi143.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-14T05:32:17.180 INFO:teuthology.orchestra.run.smithi143.stdout:Discarding blocks...Done. 2024-09-14T05:32:17.182 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi143.front.sepia.ceph.com -o noatime 2024-09-14T05:32:17.182 DEBUG:teuthology.orchestra.run.smithi143:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster1-2 2024-09-14T05:32:17.285 DEBUG:teuthology.orchestra.run.smithi143:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-2 2024-09-14T05:32:17.317 DEBUG:teuthology.orchestra.run.smithi143:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-09-14T05:32:17.412 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-14T05:32:17.411+0000 7fa0464a5640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-0/keyring: can't open /var/lib/ceph/osd/cluster1-0/keyring: (2) No such file or directory 2024-09-14T05:32:17.415 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-14T05:32:17.414+0000 7fa0464a5640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-0/keyring 2024-09-14T05:32:17.415 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-14T05:32:17.414+0000 7fa0464a5640 -1 bdev(0x559f9056f800 /var/lib/ceph/osd/cluster1-0/block) open stat got: (1) Operation not permitted 2024-09-14T05:32:17.415 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-14T05:32:17.415+0000 7fa0464a5640 -1 bluestore(/var/lib/ceph/osd/cluster1-0) _read_fsid unparsable uuid 2024-09-14T05:32:20.570 DEBUG:teuthology.orchestra.run.smithi143:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-0 2024-09-14T05:32:20.599 DEBUG:teuthology.orchestra.run.smithi143:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-09-14T05:32:20.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-14T05:32:20.689+0000 7fddb4824640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-1/keyring: can't open /var/lib/ceph/osd/cluster1-1/keyring: (2) No such file or directory 2024-09-14T05:32:20.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-14T05:32:20.689+0000 7fddb4824640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-1/keyring 2024-09-14T05:32:20.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-14T05:32:20.689+0000 7fddb4824640 -1 bdev(0x55e2ba6c9800 /var/lib/ceph/osd/cluster1-1/block) open stat got: (1) Operation not permitted 2024-09-14T05:32:20.691 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-14T05:32:20.689+0000 7fddb4824640 -1 bluestore(/var/lib/ceph/osd/cluster1-1) _read_fsid unparsable uuid 2024-09-14T05:32:23.846 DEBUG:teuthology.orchestra.run.smithi143:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-1 2024-09-14T05:32:23.872 DEBUG:teuthology.orchestra.run.smithi143:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-09-14T05:32:23.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-14T05:32:23.958+0000 7fbd51ce4640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-2/keyring: can't open /var/lib/ceph/osd/cluster1-2/keyring: (2) No such file or directory 2024-09-14T05:32:23.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-14T05:32:23.958+0000 7fbd51ce4640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-2/keyring 2024-09-14T05:32:23.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-14T05:32:23.959+0000 7fbd51ce4640 -1 bdev(0x5638278f9800 /var/lib/ceph/osd/cluster1-2/block) open stat got: (1) Operation not permitted 2024-09-14T05:32:23.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-09-14T05:32:23.959+0000 7fbd51ce4640 -1 bluestore(/var/lib/ceph/osd/cluster1-2) _read_fsid unparsable uuid 2024-09-14T05:32:26.878 DEBUG:teuthology.orchestra.run.smithi143:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-2 2024-09-14T05:32:26.907 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-14T05:32:26.907 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-14T05:32:26.907 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/mgr/cluster1-x/keyring of=/dev/stdout 2024-09-14T05:32:26.977 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-14T05:32:26.978 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/osd/cluster1-0/keyring of=/dev/stdout 2024-09-14T05:32:27.047 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-14T05:32:27.047 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/osd/cluster1-1/keyring of=/dev/stdout 2024-09-14T05:32:27.116 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-14T05:32:27.116 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/osd/cluster1-2/keyring of=/dev/stdout 2024-09-14T05:32:27.184 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-14T05:32:27.184 DEBUG:teuthology.orchestra.run.smithi143:> dd if=/etc/ceph/cluster1.client.0.keyring of=/dev/stdout 2024-09-14T05:32:27.242 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-14T05:32:27.242 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/etc/ceph/cluster1.client.mirror.keyring of=/dev/stdout 2024-09-14T05:32:27.261 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-14T05:32:27.261 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/etc/ceph/cluster1.client.mirror.0.keyring of=/dev/stdout 2024-09-14T05:32:27.320 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-14T05:32:27.321 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/etc/ceph/cluster1.client.mirror.1.keyring of=/dev/stdout 2024-09-14T05:32:27.379 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-14T05:32:27.379 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/etc/ceph/cluster1.client.mirror.2.keyring of=/dev/stdout 2024-09-14T05:32:27.437 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-14T05:32:27.437 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/etc/ceph/cluster1.client.mirror.3.keyring of=/dev/stdout 2024-09-14T05:32:27.495 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-14T05:32:27.495 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/etc/ceph/cluster1.client.mirror.4.keyring of=/dev/stdout 2024-09-14T05:32:27.553 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-14T05:32:27.553 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/etc/ceph/cluster1.client.mirror.5.keyring of=/dev/stdout 2024-09-14T05:32:27.610 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-14T05:32:27.610 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/etc/ceph/cluster1.client.mirror.6.keyring of=/dev/stdout 2024-09-14T05:32:27.668 INFO:tasks.ceph:Adding keys to all mons... 2024-09-14T05:32:27.668 DEBUG:teuthology.orchestra.run.smithi143:> sudo tee -a /etc/ceph/cluster1.keyring 2024-09-14T05:32:27.696 INFO:teuthology.orchestra.run.smithi143.stdout:[mgr.x] 2024-09-14T05:32:27.696 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQDeH+VmmttXDxAA4T/iPGrnZY9yjxtGdBWaPg== 2024-09-14T05:32:27.696 INFO:teuthology.orchestra.run.smithi143.stdout:[osd.0] 2024-09-14T05:32:27.696 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQDhH+VmCG+SGBAAGNmVCmf7cYsWZOHzccr1Cg== 2024-09-14T05:32:27.697 INFO:teuthology.orchestra.run.smithi143.stdout:[osd.1] 2024-09-14T05:32:27.697 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQDkH+VmVY8vKRAAWjTnulDJNVlq9nSryRRY1w== 2024-09-14T05:32:27.697 INFO:teuthology.orchestra.run.smithi143.stdout:[osd.2] 2024-09-14T05:32:27.697 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQDnH+Vm8xA/ORAA8Ll6G3gmuslYbsfpEAUQDA== 2024-09-14T05:32:27.697 INFO:teuthology.orchestra.run.smithi143.stdout:[client.0] 2024-09-14T05:32:27.697 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQDeH+VmN6jyERAAEp++Y3IfHDPTMxI+52sU4Q== 2024-09-14T05:32:27.697 INFO:teuthology.orchestra.run.smithi143.stdout:[client.mirror] 2024-09-14T05:32:27.697 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQDeH+Vmg2JuFhAASLB828bucYukAwh5vIqEFA== 2024-09-14T05:32:27.697 INFO:teuthology.orchestra.run.smithi143.stdout:[client.mirror.0] 2024-09-14T05:32:27.697 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQDeH+VmywrXGRAAhXLA1PXPehaSfLmKqD4ddw== 2024-09-14T05:32:27.697 INFO:teuthology.orchestra.run.smithi143.stdout:[client.mirror.1] 2024-09-14T05:32:27.697 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQDeH+VmZPaQHxAAIjyukzX7dhPkNVTP4Q0bsg== 2024-09-14T05:32:27.698 INFO:teuthology.orchestra.run.smithi143.stdout:[client.mirror.2] 2024-09-14T05:32:27.698 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQDeH+Vmngc4JRAAceWUCBxyDe0DK2ZJsSAkwA== 2024-09-14T05:32:27.698 INFO:teuthology.orchestra.run.smithi143.stdout:[client.mirror.3] 2024-09-14T05:32:27.698 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQDeH+VmGxjWKhAAVGh9ekAahsA+/81mhUOZlA== 2024-09-14T05:32:27.698 INFO:teuthology.orchestra.run.smithi143.stdout:[client.mirror.4] 2024-09-14T05:32:27.698 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQDeH+VmydFjMBAAsCgOWv4rr2wvziPnykt5wA== 2024-09-14T05:32:27.698 INFO:teuthology.orchestra.run.smithi143.stdout:[client.mirror.5] 2024-09-14T05:32:27.698 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQDeH+VmnSwDNhAA1Ghnb7hgZrFCjXo4ypM7nQ== 2024-09-14T05:32:27.698 INFO:teuthology.orchestra.run.smithi143.stdout:[client.mirror.6] 2024-09-14T05:32:27.698 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQDfH+VmV/wGABAA4V0RG2wO9Kjo2FdD2mQsAA== 2024-09-14T05:32:27.699 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-14T05:32:27.796 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-14T05:32:27.849 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-14T05:32:27.901 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-14T05:32:27.955 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-14T05:32:28.009 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-14T05:32:28.063 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-14T05:32:28.117 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-14T05:32:28.172 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-14T05:32:28.227 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-14T05:32:28.280 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-14T05:32:28.335 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-14T05:32:28.389 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-14T05:32:28.444 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-14T05:32:28.444 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /var/lib/ceph/mon/cluster1-a 2024-09-14T05:32:28.468 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster1 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster1.monmap --keyring /etc/ceph/cluster1.keyring 2024-09-14T05:32:28.565 DEBUG:teuthology.orchestra.run.smithi143:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster1-a 2024-09-14T05:32:28.590 DEBUG:teuthology.orchestra.run.smithi143:> rm -- /home/ubuntu/cephtest/cluster1.monmap 2024-09-14T05:32:28.646 INFO:tasks.ceph:Starting mon daemons in cluster cluster1... 2024-09-14T05:32:28.646 INFO:tasks.ceph.cluster1.mon.a:Restarting daemon 2024-09-14T05:32:28.646 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster1 -i a 2024-09-14T05:32:28.689 INFO:tasks.ceph.cluster1.mon.a:Started 2024-09-14T05:32:28.689 INFO:tasks.ceph:Starting mgr daemons in cluster cluster1... 2024-09-14T05:32:28.689 INFO:tasks.ceph.cluster1.mgr.x:Restarting daemon 2024-09-14T05:32:28.690 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster1 -i x 2024-09-14T05:32:28.692 INFO:tasks.ceph.cluster1.mgr.x:Started 2024-09-14T05:32:28.692 DEBUG:tasks.ceph:set 0 configs 2024-09-14T05:32:28.692 INFO:tasks.ceph:Setting crush tunables to default 2024-09-14T05:32:28.692 DEBUG:teuthology.orchestra.run.smithi143:> sudo ceph --cluster cluster1 osd crush tunables default 2024-09-14T05:32:28.866 INFO:teuthology.orchestra.run.smithi143.stderr:adjusted tunables profile to default 2024-09-14T05:32:28.877 INFO:tasks.ceph:check_enable_crimson: False 2024-09-14T05:32:28.877 INFO:tasks.ceph:Starting osd daemons in cluster cluster1... 2024-09-14T05:32:28.877 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-14T05:32:28.877 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/osd/cluster1-0/fsid of=/dev/stdout 2024-09-14T05:32:28.902 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-14T05:32:28.902 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/osd/cluster1-1/fsid of=/dev/stdout 2024-09-14T05:32:28.968 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-14T05:32:28.968 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/osd/cluster1-2/fsid of=/dev/stdout 2024-09-14T05:32:29.033 DEBUG:teuthology.orchestra.run.smithi143:> sudo ceph --cluster cluster1 osd new e7cec17d-6ff1-4ab5-8f4c-ed350a9a55cb 0 2024-09-14T05:32:29.252 INFO:teuthology.orchestra.run.smithi143.stdout:0 2024-09-14T05:32:29.263 DEBUG:teuthology.orchestra.run.smithi143:> sudo ceph --cluster cluster1 osd new f93e9b84-2f50-46a0-9a48-e6269a52ddb0 1 2024-09-14T05:32:29.440 INFO:teuthology.orchestra.run.smithi143.stdout:1 2024-09-14T05:32:29.452 DEBUG:teuthology.orchestra.run.smithi143:> sudo ceph --cluster cluster1 osd new eddc7c00-4b16-4193-afa4-ab49db2d1390 2 2024-09-14T05:32:29.633 INFO:teuthology.orchestra.run.smithi143.stdout:2 2024-09-14T05:32:29.644 INFO:tasks.ceph.cluster1.osd.0:Restarting daemon 2024-09-14T05:32:29.644 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 2024-09-14T05:32:29.647 INFO:tasks.ceph.cluster1.osd.0:Started 2024-09-14T05:32:29.647 INFO:tasks.ceph.cluster1.osd.1:Restarting daemon 2024-09-14T05:32:29.647 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 2024-09-14T05:32:29.649 INFO:tasks.ceph.cluster1.osd.1:Started 2024-09-14T05:32:29.649 INFO:tasks.ceph.cluster1.osd.2:Restarting daemon 2024-09-14T05:32:29.649 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 2024-09-14T05:32:29.652 INFO:tasks.ceph.cluster1.osd.2:Started 2024-09-14T05:32:29.652 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-14T05:32:29.829 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:29.829 INFO:teuthology.orchestra.run.smithi143.stdout:{"epoch":5,"fsid":"5628b0ee-4376-46aa-afba-0841d836b50f","created":"2024-09-14T05:32:28.774394+0000","modified":"2024-09-14T05:32:29.632378+0000","last_up_change":"0.000000","last_in_change":"2024-09-14T05:32:29.632378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7cec17d-6ff1-4ab5-8f4c-ed350a9a55cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f93e9b84-2f50-46a0-9a48-e6269a52ddb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"eddc7c00-4b16-4193-afa4-ab49db2d1390","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T05:32:29.839 INFO:tasks.ceph.ceph_manager.cluster1:[] 2024-09-14T05:32:29.840 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-14T05:32:30.142 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-09-14T05:32:30.436 INFO:teuthology.misc.health.smithi143.stderr:2024-09-14T05:32:30.435+0000 7fbf5bfff640 0 --2- 172.21.15.143:0/2834251476 >> v2:172.21.15.143:3300/0 conn(0x7fbf640a7d40 0x7fbf6414fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-14T05:32:30.504 INFO:teuthology.misc.health.smithi143.stdout: 2024-09-14T05:32:30.504 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":5,"fsid":"5628b0ee-4376-46aa-afba-0841d836b50f","created":"2024-09-14T05:32:28.774394+0000","modified":"2024-09-14T05:32:29.632378+0000","last_up_change":"0.000000","last_in_change":"2024-09-14T05:32:29.632378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7cec17d-6ff1-4ab5-8f4c-ed350a9a55cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f93e9b84-2f50-46a0-9a48-e6269a52ddb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"eddc7c00-4b16-4193-afa4-ab49db2d1390","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T05:32:30.513 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-14T05:32:30.874 INFO:tasks.ceph.cluster1.osd.2.smithi143.stderr:2024-09-14T05:32:30.873+0000 7ff5b4eb0640 -1 Falling back to public interface 2024-09-14T05:32:30.876 INFO:tasks.ceph.cluster1.osd.1.smithi143.stderr:2024-09-14T05:32:30.875+0000 7f59bf3c5640 -1 Falling back to public interface 2024-09-14T05:32:31.172 INFO:tasks.ceph.cluster1.osd.0.smithi143.stderr:2024-09-14T05:32:31.171+0000 7f891b242640 -1 Falling back to public interface 2024-09-14T05:32:34.244 INFO:tasks.ceph.cluster1.mgr.x.smithi143.stderr:2024-09-14T05:32:34.243+0000 7f53f1aa5640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-14T05:32:34.765 INFO:tasks.ceph.cluster1.osd.2.smithi143.stderr:2024-09-14T05:32:34.764+0000 7ff5b4eb0640 -1 osd.2 0 log_to_monitors true 2024-09-14T05:32:35.307 INFO:tasks.ceph.cluster1.osd.1.smithi143.stderr:2024-09-14T05:32:35.306+0000 7f59bf3c5640 -1 osd.1 0 log_to_monitors true 2024-09-14T05:32:35.338 INFO:tasks.ceph.cluster1.osd.0.smithi143.stderr:2024-09-14T05:32:35.338+0000 7f891b242640 -1 osd.0 0 log_to_monitors true 2024-09-14T05:32:36.817 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-09-14T05:32:37.001 INFO:tasks.ceph.cluster1.osd.2.smithi143.stderr:2024-09-14T05:32:37.000+0000 7ff5b0e3f640 -1 osd.2 0 waiting for initial osdmap 2024-09-14T05:32:37.014 INFO:tasks.ceph.cluster1.osd.2.smithi143.stderr:2024-09-14T05:32:37.014+0000 7ff5a8c5e640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-14T05:32:37.126 INFO:teuthology.misc.health.smithi143.stdout: 2024-09-14T05:32:37.126 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":7,"fsid":"5628b0ee-4376-46aa-afba-0841d836b50f","created":"2024-09-14T05:32:28.774394+0000","modified":"2024-09-14T05:32:36.241539+0000","last_up_change":"0.000000","last_in_change":"2024-09-14T05:32:29.632378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7cec17d-6ff1-4ab5-8f4c-ed350a9a55cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f93e9b84-2f50-46a0-9a48-e6269a52ddb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"eddc7c00-4b16-4193-afa4-ab49db2d1390","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T05:32:37.137 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-14T05:32:38.115 INFO:tasks.ceph.cluster1.osd.1.smithi143.stderr:2024-09-14T05:32:38.115+0000 7f59bb354640 -1 osd.1 0 waiting for initial osdmap 2024-09-14T05:32:38.129 INFO:tasks.ceph.cluster1.osd.0.smithi143.stderr:2024-09-14T05:32:38.128+0000 7f89171d1640 -1 osd.0 0 waiting for initial osdmap 2024-09-14T05:32:38.131 INFO:tasks.ceph.cluster1.osd.1.smithi143.stderr:2024-09-14T05:32:38.131+0000 7f59b3173640 -1 osd.1 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-14T05:32:38.143 INFO:tasks.ceph.cluster1.osd.0.smithi143.stderr:2024-09-14T05:32:38.143+0000 7f890eff0640 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-14T05:32:43.441 DEBUG:teuthology.orchestra.run.smithi143:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-09-14T05:32:43.719 INFO:teuthology.misc.health.smithi143.stdout: 2024-09-14T05:32:43.719 INFO:teuthology.misc.health.smithi143.stdout:{"epoch":13,"fsid":"5628b0ee-4376-46aa-afba-0841d836b50f","created":"2024-09-14T05:32:28.774394+0000","modified":"2024-09-14T05:32:42.269350+0000","last_up_change":"2024-09-14T05:32:38.257268+0000","last_in_change":"2024-09-14T05:32:29.632378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-14T05:32:39.405204+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7cec17d-6ff1-4ab5-8f4c-ed350a9a55cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":2308157084},{"type":"v1","addr":"172.21.15.143:6817","nonce":2308157084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":2308157084},{"type":"v1","addr":"172.21.15.143:6819","nonce":2308157084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":2308157084},{"type":"v1","addr":"172.21.15.143:6823","nonce":2308157084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":2308157084},{"type":"v1","addr":"172.21.15.143:6821","nonce":2308157084}]},"public_addr":"172.21.15.143:6817/2308157084","cluster_addr":"172.21.15.143:6819/2308157084","heartbeat_back_addr":"172.21.15.143:6823/2308157084","heartbeat_front_addr":"172.21.15.143:6821/2308157084","state":["exists","up"]},{"osd":1,"uuid":"f93e9b84-2f50-46a0-9a48-e6269a52ddb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":2580893955},{"type":"v1","addr":"172.21.15.143:6809","nonce":2580893955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":2580893955},{"type":"v1","addr":"172.21.15.143:6811","nonce":2580893955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":2580893955},{"type":"v1","addr":"172.21.15.143:6815","nonce":2580893955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":2580893955},{"type":"v1","addr":"172.21.15.143:6813","nonce":2580893955}]},"public_addr":"172.21.15.143:6809/2580893955","cluster_addr":"172.21.15.143:6811/2580893955","heartbeat_back_addr":"172.21.15.143:6815/2580893955","heartbeat_front_addr":"172.21.15.143:6813/2580893955","state":["exists","up"]},{"osd":2,"uuid":"eddc7c00-4b16-4193-afa4-ab49db2d1390","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":3394389476},{"type":"v1","addr":"172.21.15.143:6801","nonce":3394389476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":3394389476},{"type":"v1","addr":"172.21.15.143:6803","nonce":3394389476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":3394389476},{"type":"v1","addr":"172.21.15.143:6807","nonce":3394389476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":3394389476},{"type":"v1","addr":"172.21.15.143:6805","nonce":3394389476}]},"public_addr":"172.21.15.143:6801/3394389476","cluster_addr":"172.21.15.143:6803/3394389476","heartbeat_back_addr":"172.21.15.143:6807/3394389476","heartbeat_front_addr":"172.21.15.143:6805/3394389476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T05:32:36.304155+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T05:32:36.349570+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T05:32:35.787996+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T05:32:43.731 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-14T05:32:43.731 INFO:tasks.ceph:Creating RBD pool 2024-09-14T05:32:43.731 DEBUG:teuthology.orchestra.run.smithi143:> sudo ceph --cluster cluster1 osd pool create rbd 8 2024-09-14T05:32:44.284 INFO:teuthology.orchestra.run.smithi143.stderr:pool 'rbd' created 2024-09-14T05:32:44.299 DEBUG:teuthology.orchestra.run.smithi143:> rbd --cluster cluster1 pool init rbd 2024-09-14T05:32:47.305 INFO:tasks.ceph:Starting mds daemons in cluster cluster1... 2024-09-14T05:32:47.306 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 config log 1 --format=json 2024-09-14T05:32:47.307 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-14T05:32:47.601 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:47.614 INFO:teuthology.orchestra.run.smithi143.stdout:[{"version":4,"timestamp":"2024-09-14T05:32:38.129697+0000","name":"","changes":[{"name":"osd.0/osd_mclock_max_capacity_iops_ssd","new_value":"6222.429401"}]}] 2024-09-14T05:32:47.614 INFO:tasks.ceph_manager:config epoch is 4 2024-09-14T05:32:47.614 INFO:tasks.ceph:Waiting until cluster1 daemons up and pgs clean... 2024-09-14T05:32:47.615 INFO:tasks.ceph.ceph_manager.cluster1:waiting for mgr available 2024-09-14T05:32:47.615 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 mgr dump --format=json 2024-09-14T05:32:47.943 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:47.954 INFO:teuthology.orchestra.run.smithi143.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":4115184475},{"type":"v1","addr":"172.21.15.143:6825","nonce":4115184475}]},"active_addr":"172.21.15.143:6825/4115184475","active_change":"2024-09-14T05:32:33.183994+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.143:0","nonce":1728964047}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.143:0","nonce":956022391}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.143:0","nonce":919678951}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.143:0","nonce":1501597328}]}]} 2024-09-14T05:32:47.956 INFO:tasks.ceph.ceph_manager.cluster1:mgr available! 2024-09-14T05:32:47.957 INFO:tasks.ceph.ceph_manager.cluster1:waiting for all up 2024-09-14T05:32:47.957 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-14T05:32:48.239 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:48.239 INFO:teuthology.orchestra.run.smithi143.stdout:{"epoch":17,"fsid":"5628b0ee-4376-46aa-afba-0841d836b50f","created":"2024-09-14T05:32:28.774394+0000","modified":"2024-09-14T05:32:47.293605+0000","last_up_change":"2024-09-14T05:32:38.257268+0000","last_in_change":"2024-09-14T05:32:29.632378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-14T05:32:39.405204+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-14T05:32:44.015925+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7cec17d-6ff1-4ab5-8f4c-ed350a9a55cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":2308157084},{"type":"v1","addr":"172.21.15.143:6817","nonce":2308157084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":2308157084},{"type":"v1","addr":"172.21.15.143:6819","nonce":2308157084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":2308157084},{"type":"v1","addr":"172.21.15.143:6823","nonce":2308157084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":2308157084},{"type":"v1","addr":"172.21.15.143:6821","nonce":2308157084}]},"public_addr":"172.21.15.143:6817/2308157084","cluster_addr":"172.21.15.143:6819/2308157084","heartbeat_back_addr":"172.21.15.143:6823/2308157084","heartbeat_front_addr":"172.21.15.143:6821/2308157084","state":["exists","up"]},{"osd":1,"uuid":"f93e9b84-2f50-46a0-9a48-e6269a52ddb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":2580893955},{"type":"v1","addr":"172.21.15.143:6809","nonce":2580893955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":2580893955},{"type":"v1","addr":"172.21.15.143:6811","nonce":2580893955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":2580893955},{"type":"v1","addr":"172.21.15.143:6815","nonce":2580893955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":2580893955},{"type":"v1","addr":"172.21.15.143:6813","nonce":2580893955}]},"public_addr":"172.21.15.143:6809/2580893955","cluster_addr":"172.21.15.143:6811/2580893955","heartbeat_back_addr":"172.21.15.143:6815/2580893955","heartbeat_front_addr":"172.21.15.143:6813/2580893955","state":["exists","up"]},{"osd":2,"uuid":"eddc7c00-4b16-4193-afa4-ab49db2d1390","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":3394389476},{"type":"v1","addr":"172.21.15.143:6801","nonce":3394389476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":3394389476},{"type":"v1","addr":"172.21.15.143:6803","nonce":3394389476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":3394389476},{"type":"v1","addr":"172.21.15.143:6807","nonce":3394389476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":3394389476},{"type":"v1","addr":"172.21.15.143:6805","nonce":3394389476}]},"public_addr":"172.21.15.143:6801/3394389476","cluster_addr":"172.21.15.143:6803/3394389476","heartbeat_back_addr":"172.21.15.143:6807/3394389476","heartbeat_front_addr":"172.21.15.143:6805/3394389476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T05:32:36.304155+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T05:32:36.349570+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T05:32:35.787996+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T05:32:48.251 INFO:tasks.ceph.ceph_manager.cluster1:all up! 2024-09-14T05:32:48.252 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-14T05:32:48.537 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:48.537 INFO:teuthology.orchestra.run.smithi143.stdout:{"epoch":17,"fsid":"5628b0ee-4376-46aa-afba-0841d836b50f","created":"2024-09-14T05:32:28.774394+0000","modified":"2024-09-14T05:32:47.293605+0000","last_up_change":"2024-09-14T05:32:38.257268+0000","last_in_change":"2024-09-14T05:32:29.632378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-14T05:32:39.405204+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-14T05:32:44.015925+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7cec17d-6ff1-4ab5-8f4c-ed350a9a55cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":2308157084},{"type":"v1","addr":"172.21.15.143:6817","nonce":2308157084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":2308157084},{"type":"v1","addr":"172.21.15.143:6819","nonce":2308157084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":2308157084},{"type":"v1","addr":"172.21.15.143:6823","nonce":2308157084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":2308157084},{"type":"v1","addr":"172.21.15.143:6821","nonce":2308157084}]},"public_addr":"172.21.15.143:6817/2308157084","cluster_addr":"172.21.15.143:6819/2308157084","heartbeat_back_addr":"172.21.15.143:6823/2308157084","heartbeat_front_addr":"172.21.15.143:6821/2308157084","state":["exists","up"]},{"osd":1,"uuid":"f93e9b84-2f50-46a0-9a48-e6269a52ddb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":2580893955},{"type":"v1","addr":"172.21.15.143:6809","nonce":2580893955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":2580893955},{"type":"v1","addr":"172.21.15.143:6811","nonce":2580893955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":2580893955},{"type":"v1","addr":"172.21.15.143:6815","nonce":2580893955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":2580893955},{"type":"v1","addr":"172.21.15.143:6813","nonce":2580893955}]},"public_addr":"172.21.15.143:6809/2580893955","cluster_addr":"172.21.15.143:6811/2580893955","heartbeat_back_addr":"172.21.15.143:6815/2580893955","heartbeat_front_addr":"172.21.15.143:6813/2580893955","state":["exists","up"]},{"osd":2,"uuid":"eddc7c00-4b16-4193-afa4-ab49db2d1390","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":3394389476},{"type":"v1","addr":"172.21.15.143:6801","nonce":3394389476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":3394389476},{"type":"v1","addr":"172.21.15.143:6803","nonce":3394389476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":3394389476},{"type":"v1","addr":"172.21.15.143:6807","nonce":3394389476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":3394389476},{"type":"v1","addr":"172.21.15.143:6805","nonce":3394389476}]},"public_addr":"172.21.15.143:6801/3394389476","cluster_addr":"172.21.15.143:6803/3394389476","heartbeat_back_addr":"172.21.15.143:6807/3394389476","heartbeat_front_addr":"172.21.15.143:6805/3394389476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T05:32:36.304155+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T05:32:36.349570+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T05:32:35.787996+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T05:32:48.549 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 flush_pg_stats 2024-09-14T05:32:48.550 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 flush_pg_stats 2024-09-14T05:32:48.550 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 flush_pg_stats 2024-09-14T05:32:48.687 INFO:teuthology.orchestra.run.smithi143.stdout:38654705668 2024-09-14T05:32:48.687 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-09-14T05:32:48.708 INFO:teuthology.orchestra.run.smithi143.stdout:38654705668 2024-09-14T05:32:48.708 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-09-14T05:32:48.709 INFO:teuthology.orchestra.run.smithi143.stdout:34359738374 2024-09-14T05:32:48.709 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-09-14T05:32:48.998 INFO:teuthology.orchestra.run.smithi143.stdout:38654705667 2024-09-14T05:32:49.011 INFO:tasks.ceph.ceph_manager.cluster1:need seq 38654705668 got 38654705667 for osd.1 2024-09-14T05:32:49.052 INFO:teuthology.orchestra.run.smithi143.stdout:34359738372 2024-09-14T05:32:49.061 INFO:teuthology.orchestra.run.smithi143.stdout:38654705667 2024-09-14T05:32:49.064 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738374 got 34359738372 for osd.2 2024-09-14T05:32:49.074 INFO:tasks.ceph.ceph_manager.cluster1:need seq 38654705668 got 38654705667 for osd.0 2024-09-14T05:32:50.012 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-09-14T05:32:50.065 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-09-14T05:32:50.075 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-09-14T05:32:50.325 INFO:teuthology.orchestra.run.smithi143.stdout:38654705668 2024-09-14T05:32:50.338 INFO:tasks.ceph.ceph_manager.cluster1:need seq 38654705668 got 38654705668 for osd.1 2024-09-14T05:32:50.339 DEBUG:teuthology.parallel:result is None 2024-09-14T05:32:50.368 INFO:teuthology.orchestra.run.smithi143.stdout:34359738374 2024-09-14T05:32:50.381 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738374 got 34359738374 for osd.2 2024-09-14T05:32:50.381 DEBUG:teuthology.parallel:result is None 2024-09-14T05:32:50.407 INFO:teuthology.orchestra.run.smithi143.stdout:38654705668 2024-09-14T05:32:50.420 INFO:tasks.ceph.ceph_manager.cluster1:need seq 38654705668 got 38654705668 for osd.0 2024-09-14T05:32:50.420 DEBUG:teuthology.parallel:result is None 2024-09-14T05:32:50.420 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-09-14T05:32:50.420 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-14T05:32:50.737 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:50.738 INFO:teuthology.orchestra.run.smithi143.stderr:dumped all 2024-09-14T05:32:50.749 INFO:teuthology.orchestra.run.smithi143.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-14T05:32:49.190244+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81644,"kb_used_data":552,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283033876,"statfs":{"total":289910292480,"available":289826689024,"internally_reserved":0,"allocated":565248,"data_stored":1310194,"data_compressed":5974,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4783,"internal_metadata":82963793},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.905574"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-14T05:32:47.301032+0000","last_change":"2024-09-14T05:32:47.301032+0000","last_active":"2024-09-14T05:32:47.301032+0000","last_peered":"2024-09-14T05:32:47.301032+0000","last_clean":"2024-09-14T05:32:47.301032+0000","last_became_active":"2024-09-14T05:32:45.297981+0000","last_became_peered":"2024-09-14T05:32:45.297981+0000","last_unstale":"2024-09-14T05:32:47.301032+0000","last_undegraded":"2024-09-14T05:32:47.301032+0000","last_fullsized":"2024-09-14T05:32:47.301032+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_clean_scrub_stamp":"2024-09-14T05:32:44.283339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:13:28.873678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059651699999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-14T05:32:47.301315+0000","last_change":"2024-09-14T05:32:47.301315+0000","last_active":"2024-09-14T05:32:47.301315+0000","last_peered":"2024-09-14T05:32:47.301315+0000","last_clean":"2024-09-14T05:32:47.301315+0000","last_became_active":"2024-09-14T05:32:45.296192+0000","last_became_peered":"2024-09-14T05:32:45.296192+0000","last_unstale":"2024-09-14T05:32:47.301315+0000","last_undegraded":"2024-09-14T05:32:47.301315+0000","last_fullsized":"2024-09-14T05:32:47.301315+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_clean_scrub_stamp":"2024-09-14T05:32:44.283339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:48:19.029283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079042800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-14T05:32:47.301413+0000","last_change":"2024-09-14T05:32:47.301413+0000","last_active":"2024-09-14T05:32:47.301413+0000","last_peered":"2024-09-14T05:32:47.301413+0000","last_clean":"2024-09-14T05:32:47.301413+0000","last_became_active":"2024-09-14T05:32:45.296187+0000","last_became_peered":"2024-09-14T05:32:45.296187+0000","last_unstale":"2024-09-14T05:32:47.301413+0000","last_undegraded":"2024-09-14T05:32:47.301413+0000","last_fullsized":"2024-09-14T05:32:47.301413+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_clean_scrub_stamp":"2024-09-14T05:32:44.283339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:07:34.288707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089357300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-14T05:32:47.301642+0000","last_change":"2024-09-14T05:32:47.301642+0000","last_active":"2024-09-14T05:32:47.301642+0000","last_peered":"2024-09-14T05:32:47.301642+0000","last_clean":"2024-09-14T05:32:47.301642+0000","last_became_active":"2024-09-14T05:32:45.296113+0000","last_became_peered":"2024-09-14T05:32:45.296113+0000","last_unstale":"2024-09-14T05:32:47.301642+0000","last_undegraded":"2024-09-14T05:32:47.301642+0000","last_fullsized":"2024-09-14T05:32:47.301642+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_clean_scrub_stamp":"2024-09-14T05:32:44.283339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:11:57.020807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088137700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"17'2","reported_seq":22,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-14T05:32:47.299916+0000","last_change":"2024-09-14T05:32:47.298252+0000","last_active":"2024-09-14T05:32:47.299916+0000","last_peered":"2024-09-14T05:32:47.299916+0000","last_clean":"2024-09-14T05:32:47.299916+0000","last_became_active":"2024-09-14T05:32:45.298707+0000","last_became_peered":"2024-09-14T05:32:45.298707+0000","last_unstale":"2024-09-14T05:32:47.299916+0000","last_undegraded":"2024-09-14T05:32:47.299916+0000","last_fullsized":"2024-09-14T05:32:47.299916+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_clean_scrub_stamp":"2024-09-14T05:32:44.283339+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:20:40.111046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054093499999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":18,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-14T05:32:47.511782+0000","last_change":"2024-09-14T05:32:47.511782+0000","last_active":"2024-09-14T05:32:47.511782+0000","last_peered":"2024-09-14T05:32:47.511782+0000","last_clean":"2024-09-14T05:32:47.511782+0000","last_became_active":"2024-09-14T05:32:45.296961+0000","last_became_peered":"2024-09-14T05:32:45.296961+0000","last_unstale":"2024-09-14T05:32:47.511782+0000","last_undegraded":"2024-09-14T05:32:47.511782+0000","last_fullsized":"2024-09-14T05:32:47.511782+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_clean_scrub_stamp":"2024-09-14T05:32:44.283339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:49:29.999927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031355,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":18,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-14T05:32:47.511875+0000","last_change":"2024-09-14T05:32:47.511875+0000","last_active":"2024-09-14T05:32:47.511875+0000","last_peered":"2024-09-14T05:32:47.511875+0000","last_clean":"2024-09-14T05:32:47.511875+0000","last_became_active":"2024-09-14T05:32:45.298668+0000","last_became_peered":"2024-09-14T05:32:45.298668+0000","last_unstale":"2024-09-14T05:32:47.511875+0000","last_undegraded":"2024-09-14T05:32:47.511875+0000","last_fullsized":"2024-09-14T05:32:47.511875+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_clean_scrub_stamp":"2024-09-14T05:32:44.283339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T16:47:21.555137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035534800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"15'1","reported_seq":21,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-14T05:32:47.301008+0000","last_change":"2024-09-14T05:32:47.301008+0000","last_active":"2024-09-14T05:32:47.301008+0000","last_peered":"2024-09-14T05:32:47.301008+0000","last_clean":"2024-09-14T05:32:47.301008+0000","last_became_active":"2024-09-14T05:32:45.295172+0000","last_became_peered":"2024-09-14T05:32:45.295172+0000","last_unstale":"2024-09-14T05:32:47.301008+0000","last_undegraded":"2024-09-14T05:32:47.301008+0000","last_fullsized":"2024-09-14T05:32:47.301008+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_clean_scrub_stamp":"2024-09-14T05:32:44.283339+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:38:02.333228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061610699999999996,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"12'52","reported_seq":87,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-14T05:32:47.300286+0000","last_change":"2024-09-14T05:32:41.276119+0000","last_active":"2024-09-14T05:32:47.300286+0000","last_peered":"2024-09-14T05:32:47.300286+0000","last_clean":"2024-09-14T05:32:47.300286+0000","last_became_active":"2024-09-14T05:32:41.275140+0000","last_became_peered":"2024-09-14T05:32:41.275140+0000","last_unstale":"2024-09-14T05:32:47.300286+0000","last_undegraded":"2024-09-14T05:32:47.300286+0000","last_fullsized":"2024-09-14T05:32:47.300286+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:32:40.262579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:32:40.262579+0000","last_clean_scrub_stamp":"2024-09-14T05:32:40.262579+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:45:44.677928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5974,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654705668,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":208896,"data_stored":633527,"data_compressed":2987,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":208896,"data_stored":633527,"data_compressed":2987,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":147456,"data_stored":43140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2987,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2987,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T05:32:50.750 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-14T05:32:51.026 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:51.027 INFO:teuthology.orchestra.run.smithi143.stderr:dumped all 2024-09-14T05:32:51.039 INFO:teuthology.orchestra.run.smithi143.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-14T05:32:49.190244+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81644,"kb_used_data":552,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283033876,"statfs":{"total":289910292480,"available":289826689024,"internally_reserved":0,"allocated":565248,"data_stored":1310194,"data_compressed":5974,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4783,"internal_metadata":82963793},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.905574"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-14T05:32:47.301032+0000","last_change":"2024-09-14T05:32:47.301032+0000","last_active":"2024-09-14T05:32:47.301032+0000","last_peered":"2024-09-14T05:32:47.301032+0000","last_clean":"2024-09-14T05:32:47.301032+0000","last_became_active":"2024-09-14T05:32:45.297981+0000","last_became_peered":"2024-09-14T05:32:45.297981+0000","last_unstale":"2024-09-14T05:32:47.301032+0000","last_undegraded":"2024-09-14T05:32:47.301032+0000","last_fullsized":"2024-09-14T05:32:47.301032+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_clean_scrub_stamp":"2024-09-14T05:32:44.283339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:13:28.873678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059651699999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-14T05:32:47.301315+0000","last_change":"2024-09-14T05:32:47.301315+0000","last_active":"2024-09-14T05:32:47.301315+0000","last_peered":"2024-09-14T05:32:47.301315+0000","last_clean":"2024-09-14T05:32:47.301315+0000","last_became_active":"2024-09-14T05:32:45.296192+0000","last_became_peered":"2024-09-14T05:32:45.296192+0000","last_unstale":"2024-09-14T05:32:47.301315+0000","last_undegraded":"2024-09-14T05:32:47.301315+0000","last_fullsized":"2024-09-14T05:32:47.301315+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_clean_scrub_stamp":"2024-09-14T05:32:44.283339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:48:19.029283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079042800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-14T05:32:47.301413+0000","last_change":"2024-09-14T05:32:47.301413+0000","last_active":"2024-09-14T05:32:47.301413+0000","last_peered":"2024-09-14T05:32:47.301413+0000","last_clean":"2024-09-14T05:32:47.301413+0000","last_became_active":"2024-09-14T05:32:45.296187+0000","last_became_peered":"2024-09-14T05:32:45.296187+0000","last_unstale":"2024-09-14T05:32:47.301413+0000","last_undegraded":"2024-09-14T05:32:47.301413+0000","last_fullsized":"2024-09-14T05:32:47.301413+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_clean_scrub_stamp":"2024-09-14T05:32:44.283339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:07:34.288707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089357300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-14T05:32:47.301642+0000","last_change":"2024-09-14T05:32:47.301642+0000","last_active":"2024-09-14T05:32:47.301642+0000","last_peered":"2024-09-14T05:32:47.301642+0000","last_clean":"2024-09-14T05:32:47.301642+0000","last_became_active":"2024-09-14T05:32:45.296113+0000","last_became_peered":"2024-09-14T05:32:45.296113+0000","last_unstale":"2024-09-14T05:32:47.301642+0000","last_undegraded":"2024-09-14T05:32:47.301642+0000","last_fullsized":"2024-09-14T05:32:47.301642+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_clean_scrub_stamp":"2024-09-14T05:32:44.283339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:11:57.020807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088137700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"17'2","reported_seq":22,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-14T05:32:47.299916+0000","last_change":"2024-09-14T05:32:47.298252+0000","last_active":"2024-09-14T05:32:47.299916+0000","last_peered":"2024-09-14T05:32:47.299916+0000","last_clean":"2024-09-14T05:32:47.299916+0000","last_became_active":"2024-09-14T05:32:45.298707+0000","last_became_peered":"2024-09-14T05:32:45.298707+0000","last_unstale":"2024-09-14T05:32:47.299916+0000","last_undegraded":"2024-09-14T05:32:47.299916+0000","last_fullsized":"2024-09-14T05:32:47.299916+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_clean_scrub_stamp":"2024-09-14T05:32:44.283339+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:20:40.111046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054093499999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":18,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-14T05:32:47.511782+0000","last_change":"2024-09-14T05:32:47.511782+0000","last_active":"2024-09-14T05:32:47.511782+0000","last_peered":"2024-09-14T05:32:47.511782+0000","last_clean":"2024-09-14T05:32:47.511782+0000","last_became_active":"2024-09-14T05:32:45.296961+0000","last_became_peered":"2024-09-14T05:32:45.296961+0000","last_unstale":"2024-09-14T05:32:47.511782+0000","last_undegraded":"2024-09-14T05:32:47.511782+0000","last_fullsized":"2024-09-14T05:32:47.511782+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_clean_scrub_stamp":"2024-09-14T05:32:44.283339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:49:29.999927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031355,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":18,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-14T05:32:47.511875+0000","last_change":"2024-09-14T05:32:47.511875+0000","last_active":"2024-09-14T05:32:47.511875+0000","last_peered":"2024-09-14T05:32:47.511875+0000","last_clean":"2024-09-14T05:32:47.511875+0000","last_became_active":"2024-09-14T05:32:45.298668+0000","last_became_peered":"2024-09-14T05:32:45.298668+0000","last_unstale":"2024-09-14T05:32:47.511875+0000","last_undegraded":"2024-09-14T05:32:47.511875+0000","last_fullsized":"2024-09-14T05:32:47.511875+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_clean_scrub_stamp":"2024-09-14T05:32:44.283339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T16:47:21.555137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035534800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"15'1","reported_seq":21,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-14T05:32:47.301008+0000","last_change":"2024-09-14T05:32:47.301008+0000","last_active":"2024-09-14T05:32:47.301008+0000","last_peered":"2024-09-14T05:32:47.301008+0000","last_clean":"2024-09-14T05:32:47.301008+0000","last_became_active":"2024-09-14T05:32:45.295172+0000","last_became_peered":"2024-09-14T05:32:45.295172+0000","last_unstale":"2024-09-14T05:32:47.301008+0000","last_undegraded":"2024-09-14T05:32:47.301008+0000","last_fullsized":"2024-09-14T05:32:47.301008+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_clean_scrub_stamp":"2024-09-14T05:32:44.283339+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:38:02.333228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061610699999999996,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"12'52","reported_seq":87,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-14T05:32:47.300286+0000","last_change":"2024-09-14T05:32:41.276119+0000","last_active":"2024-09-14T05:32:47.300286+0000","last_peered":"2024-09-14T05:32:47.300286+0000","last_clean":"2024-09-14T05:32:47.300286+0000","last_became_active":"2024-09-14T05:32:41.275140+0000","last_became_peered":"2024-09-14T05:32:41.275140+0000","last_unstale":"2024-09-14T05:32:47.300286+0000","last_undegraded":"2024-09-14T05:32:47.300286+0000","last_fullsized":"2024-09-14T05:32:47.300286+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:32:40.262579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:32:40.262579+0000","last_clean_scrub_stamp":"2024-09-14T05:32:40.262579+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:45:44.677928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5974,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654705668,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":208896,"data_stored":633527,"data_compressed":2987,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":208896,"data_stored":633527,"data_compressed":2987,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":147456,"data_stored":43140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2987,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2987,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T05:32:51.040 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-09-14T05:32:51.040 INFO:tasks.ceph:Waiting until ceph cluster cluster1 is healthy... 2024-09-14T05:32:51.040 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy 2024-09-14T05:32:51.040 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 health --format=json 2024-09-14T05:32:51.356 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:51.356 INFO:teuthology.orchestra.run.smithi143.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-14T05:32:51.369 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy done 2024-09-14T05:32:51.369 INFO:teuthology.run_tasks:Running task ceph... 2024-09-14T05:32:51.381 INFO:tasks.ceph:Creating ceph cluster cluster2... 2024-09-14T05:32:51.381 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-14T05:32:51.381 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-10_20:16:17-rbd-main-distro-default-smithi/7900324', 'branch': 'main', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zstd policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7900324', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'teuthology-2024-09-10_20:16:17-rbd-main-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '7f24e589328ae9e639c0009d93340e5b235d0a47'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '7f24e589328ae9e639c0009d93340e5b235d0a47'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '0d61375b8ab53be4c233f5b93e463edfdff9316a'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 950, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 5839, 'sha1': '7f24e589328ae9e639c0009d93340e5b235d0a47', 'sleep_before_teardown': 0, 'subset': '99/128', 'suite': 'rbd', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_0d61375b8ab53be4c233f5b93e463edfdff9316a/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '0d61375b8ab53be4c233f5b93e463edfdff9316a', 'targets': {'smithi143.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBKp8YEKbvZo4dBiDX4/t6S8Va2K3MQZi0ygsc4Dx7PCp5DriXj2KiaYHnovMOotaCtpOJyPkhsJB6V3toaXXbo=', 'smithi191.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL2v073v2G1I34ew/oySIIhRcoJk/PRN/EB4kIRX8Gp3X0OidBnpvoZx252VCP35d73k0EnPO2qJdEPgW68DolU='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '7f24e589328ae9e639c0009d93340e5b235d0a47'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '7f24e589328ae9e639c0009d93340e5b235d0a47'}}, {'ceph': {'cluster': 'cluster2', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '7f24e589328ae9e639c0009d93340e5b235d0a47'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering,exclusive-lock,object-map,fast-diff', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_MODE': 'snapshot', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-10_20:16:17', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-14T05:32:51.382 DEBUG:teuthology.orchestra.run.smithi143:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-09-14T05:32:51.384 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-09-14T05:32:51.403 DEBUG:teuthology.orchestra.run.smithi143:> sudo install -d -m0777 -- /var/run/ceph 2024-09-14T05:32:51.439 DEBUG:teuthology.orchestra.run.smithi191:> sudo install -d -m0777 -- /var/run/ceph 2024-09-14T05:32:51.471 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-14T05:32:51.471 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/scratch_devs of=/dev/stdout 2024-09-14T05:32:51.530 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-14T05:32:51.530 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_1 2024-09-14T05:32:51.587 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-14T05:32:51.587 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-14T05:32:51.587 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-09-14T05:32:51.588 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-14T05:32:51.588 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-09-14T05:32:51.588 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-09-14 05:32:06.981417267 +0000 2024-09-14T05:32:51.588 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-09-14 05:29:16.687212775 +0000 2024-09-14T05:32:51.588 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-09-14 05:29:16.687212775 +0000 2024-09-14T05:32:51.588 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-09-14 05:29:16.687212775 +0000 2024-09-14T05:32:51.588 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-14T05:32:51.655 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-09-14T05:32:51.655 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-09-14T05:32:51.655 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000251812 s, 2.0 MB/s 2024-09-14T05:32:51.656 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-14T05:32:51.714 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_2 2024-09-14T05:32:51.771 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-14T05:32:51.771 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-14T05:32:51.772 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-14T05:32:51.772 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-14T05:32:51.772 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-09-14T05:32:51.772 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-09-14 05:32:06.981417267 +0000 2024-09-14T05:32:51.772 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-09-14 05:29:17.012207440 +0000 2024-09-14T05:32:51.772 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-09-14 05:29:17.012207440 +0000 2024-09-14T05:32:51.772 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-09-14 05:29:17.012207440 +0000 2024-09-14T05:32:51.772 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-14T05:32:51.837 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-09-14T05:32:51.837 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-09-14T05:32:51.837 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000186978 s, 2.7 MB/s 2024-09-14T05:32:51.839 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-14T05:32:51.896 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_3 2024-09-14T05:32:51.952 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-14T05:32:51.952 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-14T05:32:51.952 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-09-14T05:32:51.952 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-14T05:32:51.952 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-09-14T05:32:51.952 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-09-14 05:32:06.981417267 +0000 2024-09-14T05:32:51.952 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-09-14 05:29:17.338202089 +0000 2024-09-14T05:32:51.953 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-09-14 05:29:17.338202089 +0000 2024-09-14T05:32:51.953 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-09-14 05:29:17.338202089 +0000 2024-09-14T05:32:51.953 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-14T05:32:52.016 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-09-14T05:32:52.016 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-09-14T05:32:52.016 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000179892 s, 2.8 MB/s 2024-09-14T05:32:52.017 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-14T05:32:52.074 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_4 2024-09-14T05:32:52.129 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-14T05:32:52.129 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-14T05:32:52.129 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-09-14T05:32:52.130 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-14T05:32:52.130 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-09-14T05:32:52.130 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-09-14 05:32:06.982417251 +0000 2024-09-14T05:32:52.130 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-09-14 05:29:17.677196524 +0000 2024-09-14T05:32:52.130 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-09-14 05:29:17.677196524 +0000 2024-09-14T05:32:52.130 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-09-14 05:29:17.677196524 +0000 2024-09-14T05:32:52.130 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-14T05:32:52.193 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-09-14T05:32:52.193 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-09-14T05:32:52.193 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.00020359 s, 2.5 MB/s 2024-09-14T05:32:52.194 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-14T05:32:52.251 INFO:tasks.ceph:osd dev map: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-14T05:32:52.251 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi191.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-14T05:32:52.251 INFO:tasks.ceph:Generating config... 2024-09-14T05:32:52.252 INFO:tasks.ceph:[client] debug rbd = 20 2024-09-14T05:32:52.252 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-09-14T05:32:52.252 INFO:tasks.ceph:[client] log to stderr = False 2024-09-14T05:32:52.253 INFO:tasks.ceph:[client] rbd default features = 125 2024-09-14T05:32:52.253 INFO:tasks.ceph:[client] rbd mirror image policy type = none 2024-09-14T05:32:52.253 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T05:32:52.253 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T05:32:52.253 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T05:32:52.253 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T05:32:52.253 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T05:32:52.253 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T05:32:52.253 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T05:32:52.253 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T05:32:52.253 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-14T05:32:52.253 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-09-14T05:32:52.253 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-14T05:32:52.254 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-14T05:32:52.254 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-14T05:32:52.254 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-14T05:32:52.254 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-14T05:32:52.254 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-14T05:32:52.254 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-14T05:32:52.254 INFO:tasks.ceph:[osd] bluestore compression algorithm = zstd 2024-09-14T05:32:52.254 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-14T05:32:52.254 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-14T05:32:52.254 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-14T05:32:52.254 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-14T05:32:52.254 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-14T05:32:52.254 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-14T05:32:52.254 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-14T05:32:52.255 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-14T05:32:52.255 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-14T05:32:52.255 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-14T05:32:52.255 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-14T05:32:52.255 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-14T05:32:52.255 INFO:tasks.ceph:Setting up cluster2.mon.a... 2024-09-14T05:32:52.255 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster2.keyring 2024-09-14T05:32:52.332 INFO:teuthology.orchestra.run.smithi191.stdout:creating /etc/ceph/cluster2.keyring 2024-09-14T05:32:52.335 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster2.keyring 2024-09-14T05:32:52.418 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-09-14T05:32:52.483 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster2.mon.a', '172.21.15.191')] 2024-09-14T05:32:52.484 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.191', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-09-14T05:32:52.485 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-14T05:32:52.485 DEBUG:teuthology.orchestra.run.smithi191:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-14T05:32:52.539 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.191 --print /home/ubuntu/cephtest/cluster2.monmap 2024-09-14T05:32:52.620 INFO:teuthology.orchestra.run.smithi191.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster2.monmap 2024-09-14T05:32:52.620 INFO:teuthology.orchestra.run.smithi191.stdout:monmaptool: generated fsid 2fb493c9-6b9e-47da-a932-4884bef35060 2024-09-14T05:32:52.620 INFO:teuthology.orchestra.run.smithi191.stdout:setting min_mon_release = quincy 2024-09-14T05:32:52.620 INFO:teuthology.orchestra.run.smithi191.stdout:epoch 0 2024-09-14T05:32:52.620 INFO:teuthology.orchestra.run.smithi191.stdout:fsid 2fb493c9-6b9e-47da-a932-4884bef35060 2024-09-14T05:32:52.620 INFO:teuthology.orchestra.run.smithi191.stdout:last_changed 2024-09-14T05:32:52.620161+0000 2024-09-14T05:32:52.620 INFO:teuthology.orchestra.run.smithi191.stdout:created 2024-09-14T05:32:52.620161+0000 2024-09-14T05:32:52.620 INFO:teuthology.orchestra.run.smithi191.stdout:min_mon_release 17 (quincy) 2024-09-14T05:32:52.620 INFO:teuthology.orchestra.run.smithi191.stdout:election_strategy: 1 2024-09-14T05:32:52.621 INFO:teuthology.orchestra.run.smithi191.stdout:0: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.a 2024-09-14T05:32:52.621 INFO:teuthology.orchestra.run.smithi191.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster2.monmap (1 monitors) 2024-09-14T05:32:52.622 DEBUG:teuthology.orchestra.run.smithi191:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-14T05:32:52.676 INFO:tasks.ceph:Writing /etc/ceph/cluster2.conf for FSID 2fb493c9-6b9e-47da-a932-4884bef35060... 2024-09-14T05:32:52.677 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-09-14T05:32:52.680 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-09-14T05:32:52.728 INFO:teuthology.orchestra.run.smithi143.stdout:[global] 2024-09-14T05:32:52.728 INFO:teuthology.orchestra.run.smithi143.stdout: chdir = "" 2024-09-14T05:32:52.728 INFO:teuthology.orchestra.run.smithi143.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-14T05:32:52.728 INFO:teuthology.orchestra.run.smithi143.stdout: auth supported = cephx 2024-09-14T05:32:52.728 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:52.728 INFO:teuthology.orchestra.run.smithi143.stdout: filestore xattr use omap = true 2024-09-14T05:32:52.728 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:52.728 INFO:teuthology.orchestra.run.smithi143.stdout: mon clock drift allowed = 1.000 2024-09-14T05:32:52.728 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:52.728 INFO:teuthology.orchestra.run.smithi143.stdout: osd crush chooseleaf type = 0 2024-09-14T05:32:52.729 INFO:teuthology.orchestra.run.smithi143.stdout: auth debug = true 2024-09-14T05:32:52.729 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:52.729 INFO:teuthology.orchestra.run.smithi143.stdout: ms die on old message = true 2024-09-14T05:32:52.729 INFO:teuthology.orchestra.run.smithi143.stdout: ms die on bug = true 2024-09-14T05:32:52.729 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:52.729 INFO:teuthology.orchestra.run.smithi143.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-14T05:32:52.729 INFO:teuthology.orchestra.run.smithi143.stdout: mon pg warn max object skew = 0 2024-09-14T05:32:52.729 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:52.729 INFO:teuthology.orchestra.run.smithi143.stdout: # disable pg_autoscaler by default for new pools 2024-09-14T05:32:52.729 INFO:teuthology.orchestra.run.smithi143.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-14T05:32:52.729 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:52.729 INFO:teuthology.orchestra.run.smithi143.stdout: osd pool default size = 2 2024-09-14T05:32:52.729 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:52.730 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd allow primary affinity = true 2024-09-14T05:32:52.730 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd allow pg remap = true 2024-09-14T05:32:52.730 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on legacy crush tunables = false 2024-09-14T05:32:52.730 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on crush straw calc version zero = false 2024-09-14T05:32:52.730 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on no sortbitwise = false 2024-09-14T05:32:52.730 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on osd down out interval zero = false 2024-09-14T05:32:52.730 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on too few osds = false 2024-09-14T05:32:52.730 INFO:teuthology.orchestra.run.smithi143.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-14T05:32:52.730 INFO:teuthology.orchestra.run.smithi143.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-14T05:32:52.730 INFO:teuthology.orchestra.run.smithi143.stdout: mon_allow_pool_size_one = true 2024-09-14T05:32:52.730 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:52.730 INFO:teuthology.orchestra.run.smithi143.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-14T05:32:52.730 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:52.731 INFO:teuthology.orchestra.run.smithi143.stdout: osd default data pool replay window = 5 2024-09-14T05:32:52.731 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:52.731 INFO:teuthology.orchestra.run.smithi143.stdout: mon allow pool delete = true 2024-09-14T05:32:52.731 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:52.731 INFO:teuthology.orchestra.run.smithi143.stdout: mon cluster log file level = debug 2024-09-14T05:32:52.731 INFO:teuthology.orchestra.run.smithi143.stdout: debug asserts on shutdown = true 2024-09-14T05:32:52.731 INFO:teuthology.orchestra.run.smithi143.stdout: mon health detail to clog = false 2024-09-14T05:32:52.731 INFO:teuthology.orchestra.run.smithi143.stdout: mon host = 172.21.15.191 2024-09-14T05:32:52.731 INFO:teuthology.orchestra.run.smithi143.stdout: mon client directed command retry = 5 2024-09-14T05:32:52.731 INFO:teuthology.orchestra.run.smithi143.stdout: mon warn on pool no app = False 2024-09-14T05:32:52.731 INFO:teuthology.orchestra.run.smithi143.stdout: ms inject socket failures = 5000 2024-09-14T05:32:52.731 INFO:teuthology.orchestra.run.smithi143.stdout: fsid = 2fb493c9-6b9e-47da-a932-4884bef35060 2024-09-14T05:32:52.731 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:52.731 INFO:teuthology.orchestra.run.smithi143.stdout:[osd] 2024-09-14T05:32:52.732 INFO:teuthology.orchestra.run.smithi143.stdout: osd journal size = 100 2024-09-14T05:32:52.732 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:52.732 INFO:teuthology.orchestra.run.smithi143.stdout: osd scrub load threshold = 5.0 2024-09-14T05:32:52.732 INFO:teuthology.orchestra.run.smithi143.stdout: osd scrub max interval = 600 2024-09-14T05:32:52.732 INFO:teuthology.orchestra.run.smithi143.stdout: osd mclock profile = high_recovery_ops 2024-09-14T05:32:52.732 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:52.732 INFO:teuthology.orchestra.run.smithi143.stdout: osd recover clone overlap = true 2024-09-14T05:32:52.732 INFO:teuthology.orchestra.run.smithi143.stdout: osd recovery max chunk = 1048576 2024-09-14T05:32:52.732 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:52.732 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug shutdown = true 2024-09-14T05:32:52.732 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug op order = true 2024-09-14T05:32:52.732 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug verify stray on activate = true 2024-09-14T05:32:52.732 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug trim objects = true 2024-09-14T05:32:52.732 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:52.733 INFO:teuthology.orchestra.run.smithi143.stdout: osd open classes on start = true 2024-09-14T05:32:52.733 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug pg log writeout = true 2024-09-14T05:32:52.733 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:52.733 INFO:teuthology.orchestra.run.smithi143.stdout: osd deep scrub update digest min age = 30 2024-09-14T05:32:52.733 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:52.733 INFO:teuthology.orchestra.run.smithi143.stdout: osd map max advance = 10 2024-09-14T05:32:52.733 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:52.733 INFO:teuthology.orchestra.run.smithi143.stdout: journal zero on create = true 2024-09-14T05:32:52.733 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:52.733 INFO:teuthology.orchestra.run.smithi143.stdout: filestore ondisk finisher threads = 3 2024-09-14T05:32:52.733 INFO:teuthology.orchestra.run.smithi143.stdout: filestore apply finisher threads = 3 2024-09-14T05:32:52.733 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:52.733 INFO:teuthology.orchestra.run.smithi143.stdout: bdev debug aio = true 2024-09-14T05:32:52.734 INFO:teuthology.orchestra.run.smithi143.stdout: osd debug misdirected ops = true 2024-09-14T05:32:52.734 INFO:teuthology.orchestra.run.smithi143.stdout: bluestore block size = 96636764160 2024-09-14T05:32:52.734 INFO:teuthology.orchestra.run.smithi143.stdout: bluestore compression algorithm = zstd 2024-09-14T05:32:52.734 INFO:teuthology.orchestra.run.smithi143.stdout: bluestore compression mode = aggressive 2024-09-14T05:32:52.734 INFO:teuthology.orchestra.run.smithi143.stdout: bluestore fsck on mount = True 2024-09-14T05:32:52.734 INFO:teuthology.orchestra.run.smithi143.stdout: debug bluefs = 1/20 2024-09-14T05:32:52.734 INFO:teuthology.orchestra.run.smithi143.stdout: debug bluestore = 1/20 2024-09-14T05:32:52.734 INFO:teuthology.orchestra.run.smithi143.stdout: debug ms = 1 2024-09-14T05:32:52.734 INFO:teuthology.orchestra.run.smithi143.stdout: debug osd = 20 2024-09-14T05:32:52.734 INFO:teuthology.orchestra.run.smithi143.stdout: debug rocksdb = 4/10 2024-09-14T05:32:52.734 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd backfillfull_ratio = 0.85 2024-09-14T05:32:52.734 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd full ratio = 0.9 2024-09-14T05:32:52.734 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd nearfull ratio = 0.8 2024-09-14T05:32:52.734 INFO:teuthology.orchestra.run.smithi143.stdout: osd failsafe full ratio = 0.95 2024-09-14T05:32:52.734 INFO:teuthology.orchestra.run.smithi143.stdout: osd objectstore = bluestore 2024-09-14T05:32:52.735 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:52.735 INFO:teuthology.orchestra.run.smithi143.stdout:[mgr] 2024-09-14T05:32:52.735 INFO:teuthology.orchestra.run.smithi143.stdout: debug ms = 1 2024-09-14T05:32:52.735 INFO:teuthology.orchestra.run.smithi143.stdout: debug mgr = 20 2024-09-14T05:32:52.735 INFO:teuthology.orchestra.run.smithi143.stdout: debug mon = 20 2024-09-14T05:32:52.735 INFO:teuthology.orchestra.run.smithi143.stdout: debug auth = 20 2024-09-14T05:32:52.735 INFO:teuthology.orchestra.run.smithi143.stdout: mon reweight min pgs per osd = 4 2024-09-14T05:32:52.735 INFO:teuthology.orchestra.run.smithi143.stdout: mon reweight min bytes per osd = 10 2024-09-14T05:32:52.735 INFO:teuthology.orchestra.run.smithi143.stdout: mgr/telemetry/nag = false 2024-09-14T05:32:52.735 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:52.735 INFO:teuthology.orchestra.run.smithi143.stdout:[mon] 2024-09-14T05:32:52.735 INFO:teuthology.orchestra.run.smithi143.stdout: debug ms = 1 2024-09-14T05:32:52.735 INFO:teuthology.orchestra.run.smithi143.stdout: debug mon = 20 2024-09-14T05:32:52.735 INFO:teuthology.orchestra.run.smithi143.stdout: debug paxos = 20 2024-09-14T05:32:52.736 INFO:teuthology.orchestra.run.smithi143.stdout: debug auth = 20 2024-09-14T05:32:52.736 INFO:teuthology.orchestra.run.smithi143.stdout: mon data avail warn = 5 2024-09-14T05:32:52.736 INFO:teuthology.orchestra.run.smithi143.stdout: mon mgr mkfs grace = 240 2024-09-14T05:32:52.736 INFO:teuthology.orchestra.run.smithi143.stdout: mon reweight min pgs per osd = 4 2024-09-14T05:32:52.736 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd reporter subtree level = osd 2024-09-14T05:32:52.736 INFO:teuthology.orchestra.run.smithi143.stdout: mon osd prime pg temp = true 2024-09-14T05:32:52.736 INFO:teuthology.orchestra.run.smithi143.stdout: mon reweight min bytes per osd = 10 2024-09-14T05:32:52.736 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:52.736 INFO:teuthology.orchestra.run.smithi143.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-14T05:32:52.736 INFO:teuthology.orchestra.run.smithi143.stdout: auth mon ticket ttl = 660 # 11m 2024-09-14T05:32:52.736 INFO:teuthology.orchestra.run.smithi143.stdout: auth service ticket ttl = 240 # 4m 2024-09-14T05:32:52.736 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:52.736 INFO:teuthology.orchestra.run.smithi143.stdout: # don't complain about insecure global_id in the test suite 2024-09-14T05:32:52.736 INFO:teuthology.orchestra.run.smithi143.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-14T05:32:52.737 INFO:teuthology.orchestra.run.smithi143.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-14T05:32:52.737 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:52.737 INFO:teuthology.orchestra.run.smithi143.stdout: # 1m isn't quite enough 2024-09-14T05:32:52.737 INFO:teuthology.orchestra.run.smithi143.stdout: mon_down_mkfs_grace = 2m 2024-09-14T05:32:52.737 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:52.737 INFO:teuthology.orchestra.run.smithi143.stdout: mon_warn_on_filestore_osds = false 2024-09-14T05:32:52.737 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T05:32:52.737 INFO:teuthology.orchestra.run.smithi143.stdout:[client] 2024-09-14T05:32:52.737 INFO:teuthology.orchestra.run.smithi143.stdout: rgw cache enabled = true 2024-09-14T05:32:52.737 INFO:teuthology.orchestra.run.smithi143.stdout: rgw enable ops log = true 2024-09-14T05:32:52.737 INFO:teuthology.orchestra.run.smithi143.stdout: rgw enable usage log = true 2024-09-14T05:32:52.737 INFO:teuthology.orchestra.run.smithi143.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-14T05:32:52.737 INFO:teuthology.orchestra.run.smithi143.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-14T05:32:52.737 INFO:teuthology.orchestra.run.smithi143.stdout: debug rbd = 20 2024-09-14T05:32:52.738 INFO:teuthology.orchestra.run.smithi143.stdout: debug rbd_mirror = 15 2024-09-14T05:32:52.738 INFO:teuthology.orchestra.run.smithi143.stdout: log to stderr = False 2024-09-14T05:32:52.738 INFO:teuthology.orchestra.run.smithi143.stdout: rbd default features = 125 2024-09-14T05:32:52.738 INFO:teuthology.orchestra.run.smithi143.stdout: rbd mirror image policy type = none 2024-09-14T05:32:52.738 INFO:teuthology.orchestra.run.smithi143.stdout:[mon.a] 2024-09-14T05:32:52.738 INFO:teuthology.orchestra.run.smithi143.stdout:[client.mirror.0] 2024-09-14T05:32:52.738 INFO:teuthology.orchestra.run.smithi143.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T05:32:52.738 INFO:teuthology.orchestra.run.smithi143.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T05:32:52.738 INFO:teuthology.orchestra.run.smithi143.stdout:[client.mirror.1] 2024-09-14T05:32:52.738 INFO:teuthology.orchestra.run.smithi143.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T05:32:52.738 INFO:teuthology.orchestra.run.smithi143.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T05:32:52.738 INFO:teuthology.orchestra.run.smithi143.stdout:[client.mirror.2] 2024-09-14T05:32:52.738 INFO:teuthology.orchestra.run.smithi143.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T05:32:52.738 INFO:teuthology.orchestra.run.smithi143.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T05:32:52.739 INFO:teuthology.orchestra.run.smithi143.stdout:[client.mirror.3] 2024-09-14T05:32:52.739 INFO:teuthology.orchestra.run.smithi143.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T05:32:52.739 INFO:teuthology.orchestra.run.smithi143.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T05:32:52.759 INFO:teuthology.orchestra.run.smithi191.stdout:[global] 2024-09-14T05:32:52.759 INFO:teuthology.orchestra.run.smithi191.stdout: chdir = "" 2024-09-14T05:32:52.760 INFO:teuthology.orchestra.run.smithi191.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-14T05:32:52.760 INFO:teuthology.orchestra.run.smithi191.stdout: auth supported = cephx 2024-09-14T05:32:52.760 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:52.760 INFO:teuthology.orchestra.run.smithi191.stdout: filestore xattr use omap = true 2024-09-14T05:32:52.760 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:52.760 INFO:teuthology.orchestra.run.smithi191.stdout: mon clock drift allowed = 1.000 2024-09-14T05:32:52.760 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:52.760 INFO:teuthology.orchestra.run.smithi191.stdout: osd crush chooseleaf type = 0 2024-09-14T05:32:52.760 INFO:teuthology.orchestra.run.smithi191.stdout: auth debug = true 2024-09-14T05:32:52.760 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:52.760 INFO:teuthology.orchestra.run.smithi191.stdout: ms die on old message = true 2024-09-14T05:32:52.760 INFO:teuthology.orchestra.run.smithi191.stdout: ms die on bug = true 2024-09-14T05:32:52.761 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:52.761 INFO:teuthology.orchestra.run.smithi191.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-14T05:32:52.761 INFO:teuthology.orchestra.run.smithi191.stdout: mon pg warn max object skew = 0 2024-09-14T05:32:52.761 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:52.761 INFO:teuthology.orchestra.run.smithi191.stdout: # disable pg_autoscaler by default for new pools 2024-09-14T05:32:52.761 INFO:teuthology.orchestra.run.smithi191.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-14T05:32:52.761 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:52.761 INFO:teuthology.orchestra.run.smithi191.stdout: osd pool default size = 2 2024-09-14T05:32:52.761 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:52.761 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd allow primary affinity = true 2024-09-14T05:32:52.761 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd allow pg remap = true 2024-09-14T05:32:52.761 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on legacy crush tunables = false 2024-09-14T05:32:52.761 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on crush straw calc version zero = false 2024-09-14T05:32:52.761 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on no sortbitwise = false 2024-09-14T05:32:52.762 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on osd down out interval zero = false 2024-09-14T05:32:52.762 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on too few osds = false 2024-09-14T05:32:52.762 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-14T05:32:52.762 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-14T05:32:52.762 INFO:teuthology.orchestra.run.smithi191.stdout: mon_allow_pool_size_one = true 2024-09-14T05:32:52.762 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:52.762 INFO:teuthology.orchestra.run.smithi191.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-14T05:32:52.762 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:52.762 INFO:teuthology.orchestra.run.smithi191.stdout: osd default data pool replay window = 5 2024-09-14T05:32:52.762 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:52.762 INFO:teuthology.orchestra.run.smithi191.stdout: mon allow pool delete = true 2024-09-14T05:32:52.762 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:52.762 INFO:teuthology.orchestra.run.smithi191.stdout: mon cluster log file level = debug 2024-09-14T05:32:52.762 INFO:teuthology.orchestra.run.smithi191.stdout: debug asserts on shutdown = true 2024-09-14T05:32:52.762 INFO:teuthology.orchestra.run.smithi191.stdout: mon health detail to clog = false 2024-09-14T05:32:52.763 INFO:teuthology.orchestra.run.smithi191.stdout: mon host = 172.21.15.191 2024-09-14T05:32:52.763 INFO:teuthology.orchestra.run.smithi191.stdout: mon client directed command retry = 5 2024-09-14T05:32:52.763 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on pool no app = False 2024-09-14T05:32:52.763 INFO:teuthology.orchestra.run.smithi191.stdout: ms inject socket failures = 5000 2024-09-14T05:32:52.763 INFO:teuthology.orchestra.run.smithi191.stdout: fsid = 2fb493c9-6b9e-47da-a932-4884bef35060 2024-09-14T05:32:52.763 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:52.763 INFO:teuthology.orchestra.run.smithi191.stdout:[osd] 2024-09-14T05:32:52.763 INFO:teuthology.orchestra.run.smithi191.stdout: osd journal size = 100 2024-09-14T05:32:52.763 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:52.763 INFO:teuthology.orchestra.run.smithi191.stdout: osd scrub load threshold = 5.0 2024-09-14T05:32:52.763 INFO:teuthology.orchestra.run.smithi191.stdout: osd scrub max interval = 600 2024-09-14T05:32:52.763 INFO:teuthology.orchestra.run.smithi191.stdout: osd mclock profile = high_recovery_ops 2024-09-14T05:32:52.763 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:52.764 INFO:teuthology.orchestra.run.smithi191.stdout: osd recover clone overlap = true 2024-09-14T05:32:52.764 INFO:teuthology.orchestra.run.smithi191.stdout: osd recovery max chunk = 1048576 2024-09-14T05:32:52.764 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:52.764 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug shutdown = true 2024-09-14T05:32:52.764 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug op order = true 2024-09-14T05:32:52.764 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug verify stray on activate = true 2024-09-14T05:32:52.764 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug trim objects = true 2024-09-14T05:32:52.764 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:52.764 INFO:teuthology.orchestra.run.smithi191.stdout: osd open classes on start = true 2024-09-14T05:32:52.764 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug pg log writeout = true 2024-09-14T05:32:52.764 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:52.764 INFO:teuthology.orchestra.run.smithi191.stdout: osd deep scrub update digest min age = 30 2024-09-14T05:32:52.764 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:52.764 INFO:teuthology.orchestra.run.smithi191.stdout: osd map max advance = 10 2024-09-14T05:32:52.765 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:52.765 INFO:teuthology.orchestra.run.smithi191.stdout: journal zero on create = true 2024-09-14T05:32:52.765 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:52.765 INFO:teuthology.orchestra.run.smithi191.stdout: filestore ondisk finisher threads = 3 2024-09-14T05:32:52.765 INFO:teuthology.orchestra.run.smithi191.stdout: filestore apply finisher threads = 3 2024-09-14T05:32:52.765 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:52.765 INFO:teuthology.orchestra.run.smithi191.stdout: bdev debug aio = true 2024-09-14T05:32:52.765 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug misdirected ops = true 2024-09-14T05:32:52.765 INFO:teuthology.orchestra.run.smithi191.stdout: bluestore block size = 96636764160 2024-09-14T05:32:52.765 INFO:teuthology.orchestra.run.smithi191.stdout: bluestore compression algorithm = zstd 2024-09-14T05:32:52.765 INFO:teuthology.orchestra.run.smithi191.stdout: bluestore compression mode = aggressive 2024-09-14T05:32:52.765 INFO:teuthology.orchestra.run.smithi191.stdout: bluestore fsck on mount = True 2024-09-14T05:32:52.765 INFO:teuthology.orchestra.run.smithi191.stdout: debug bluefs = 1/20 2024-09-14T05:32:52.766 INFO:teuthology.orchestra.run.smithi191.stdout: debug bluestore = 1/20 2024-09-14T05:32:52.766 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2024-09-14T05:32:52.766 INFO:teuthology.orchestra.run.smithi191.stdout: debug osd = 20 2024-09-14T05:32:52.766 INFO:teuthology.orchestra.run.smithi191.stdout: debug rocksdb = 4/10 2024-09-14T05:32:52.766 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd backfillfull_ratio = 0.85 2024-09-14T05:32:52.766 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd full ratio = 0.9 2024-09-14T05:32:52.766 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd nearfull ratio = 0.8 2024-09-14T05:32:52.766 INFO:teuthology.orchestra.run.smithi191.stdout: osd failsafe full ratio = 0.95 2024-09-14T05:32:52.766 INFO:teuthology.orchestra.run.smithi191.stdout: osd objectstore = bluestore 2024-09-14T05:32:52.766 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:52.766 INFO:teuthology.orchestra.run.smithi191.stdout:[mgr] 2024-09-14T05:32:52.766 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2024-09-14T05:32:52.766 INFO:teuthology.orchestra.run.smithi191.stdout: debug mgr = 20 2024-09-14T05:32:52.766 INFO:teuthology.orchestra.run.smithi191.stdout: debug mon = 20 2024-09-14T05:32:52.767 INFO:teuthology.orchestra.run.smithi191.stdout: debug auth = 20 2024-09-14T05:32:52.767 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min pgs per osd = 4 2024-09-14T05:32:52.767 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min bytes per osd = 10 2024-09-14T05:32:52.767 INFO:teuthology.orchestra.run.smithi191.stdout: mgr/telemetry/nag = false 2024-09-14T05:32:52.767 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:52.767 INFO:teuthology.orchestra.run.smithi191.stdout:[mon] 2024-09-14T05:32:52.767 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2024-09-14T05:32:52.767 INFO:teuthology.orchestra.run.smithi191.stdout: debug mon = 20 2024-09-14T05:32:52.767 INFO:teuthology.orchestra.run.smithi191.stdout: debug paxos = 20 2024-09-14T05:32:52.767 INFO:teuthology.orchestra.run.smithi191.stdout: debug auth = 20 2024-09-14T05:32:52.767 INFO:teuthology.orchestra.run.smithi191.stdout: mon data avail warn = 5 2024-09-14T05:32:52.767 INFO:teuthology.orchestra.run.smithi191.stdout: mon mgr mkfs grace = 240 2024-09-14T05:32:52.767 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min pgs per osd = 4 2024-09-14T05:32:52.767 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd reporter subtree level = osd 2024-09-14T05:32:52.767 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd prime pg temp = true 2024-09-14T05:32:52.768 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min bytes per osd = 10 2024-09-14T05:32:52.768 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:52.768 INFO:teuthology.orchestra.run.smithi191.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-14T05:32:52.768 INFO:teuthology.orchestra.run.smithi191.stdout: auth mon ticket ttl = 660 # 11m 2024-09-14T05:32:52.768 INFO:teuthology.orchestra.run.smithi191.stdout: auth service ticket ttl = 240 # 4m 2024-09-14T05:32:52.768 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:52.768 INFO:teuthology.orchestra.run.smithi191.stdout: # don't complain about insecure global_id in the test suite 2024-09-14T05:32:52.768 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-14T05:32:52.768 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-14T05:32:52.768 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:52.768 INFO:teuthology.orchestra.run.smithi191.stdout: # 1m isn't quite enough 2024-09-14T05:32:52.768 INFO:teuthology.orchestra.run.smithi191.stdout: mon_down_mkfs_grace = 2m 2024-09-14T05:32:52.768 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:52.769 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_filestore_osds = false 2024-09-14T05:32:52.769 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:32:52.769 INFO:teuthology.orchestra.run.smithi191.stdout:[client] 2024-09-14T05:32:52.769 INFO:teuthology.orchestra.run.smithi191.stdout: rgw cache enabled = true 2024-09-14T05:32:52.769 INFO:teuthology.orchestra.run.smithi191.stdout: rgw enable ops log = true 2024-09-14T05:32:52.769 INFO:teuthology.orchestra.run.smithi191.stdout: rgw enable usage log = true 2024-09-14T05:32:52.769 INFO:teuthology.orchestra.run.smithi191.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-14T05:32:52.769 INFO:teuthology.orchestra.run.smithi191.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-14T05:32:52.769 INFO:teuthology.orchestra.run.smithi191.stdout: debug rbd = 20 2024-09-14T05:32:52.769 INFO:teuthology.orchestra.run.smithi191.stdout: debug rbd_mirror = 15 2024-09-14T05:32:52.769 INFO:teuthology.orchestra.run.smithi191.stdout: log to stderr = False 2024-09-14T05:32:52.769 INFO:teuthology.orchestra.run.smithi191.stdout: rbd default features = 125 2024-09-14T05:32:52.769 INFO:teuthology.orchestra.run.smithi191.stdout: rbd mirror image policy type = none 2024-09-14T05:32:52.769 INFO:teuthology.orchestra.run.smithi191.stdout:[mon.a] 2024-09-14T05:32:52.770 INFO:teuthology.orchestra.run.smithi191.stdout:[client.mirror.0] 2024-09-14T05:32:52.770 INFO:teuthology.orchestra.run.smithi191.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T05:32:52.770 INFO:teuthology.orchestra.run.smithi191.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T05:32:52.770 INFO:teuthology.orchestra.run.smithi191.stdout:[client.mirror.1] 2024-09-14T05:32:52.770 INFO:teuthology.orchestra.run.smithi191.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T05:32:52.770 INFO:teuthology.orchestra.run.smithi191.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T05:32:52.770 INFO:teuthology.orchestra.run.smithi191.stdout:[client.mirror.2] 2024-09-14T05:32:52.770 INFO:teuthology.orchestra.run.smithi191.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T05:32:52.770 INFO:teuthology.orchestra.run.smithi191.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T05:32:52.770 INFO:teuthology.orchestra.run.smithi191.stdout:[client.mirror.3] 2024-09-14T05:32:52.770 INFO:teuthology.orchestra.run.smithi191.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-14T05:32:52.770 INFO:teuthology.orchestra.run.smithi191.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-14T05:32:52.771 INFO:tasks.ceph:Creating admin key on cluster2.mon.a... 2024-09-14T05:32:52.771 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster2.keyring 2024-09-14T05:32:52.866 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-14T05:32:52.866 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-14T05:32:52.867 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/etc/ceph/cluster2.keyring of=/dev/stdout 2024-09-14T05:32:52.883 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-14T05:32:52.883 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/home/ubuntu/cephtest/cluster2.monmap of=/dev/stdout 2024-09-14T05:32:52.937 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi143.front.sepia.ceph.com 2024-09-14T05:32:52.937 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-14T05:32:52.938 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/etc/ceph/cluster2.keyring 2024-09-14T05:32:52.938 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-09-14T05:32:52.974 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-14T05:32:52.974 DEBUG:teuthology.orchestra.run.smithi143:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-09-14T05:32:53.030 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi191.front.sepia.ceph.com 2024-09-14T05:32:53.030 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-14T05:32:53.030 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/ceph/cluster2.keyring 2024-09-14T05:32:53.031 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-09-14T05:32:53.065 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-14T05:32:53.065 DEBUG:teuthology.orchestra.run.smithi191:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-09-14T05:32:53.122 INFO:tasks.ceph:Setting up mon nodes... 2024-09-14T05:32:53.122 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-14T05:32:53.122 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /var/lib/ceph/mgr/cluster2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster2-x/keyring 2024-09-14T05:32:53.173 INFO:teuthology.orchestra.run.smithi143.stdout:creating /var/lib/ceph/mgr/cluster2-x/keyring 2024-09-14T05:32:53.176 INFO:tasks.ceph:Setting up mds nodes... 2024-09-14T05:32:53.176 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-14T05:32:53.176 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster2.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.0.keyring 2024-09-14T05:32:53.217 INFO:teuthology.orchestra.run.smithi143.stdout:creating /etc/ceph/cluster2.client.0.keyring 2024-09-14T05:32:53.230 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster2.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.keyring 2024-09-14T05:32:53.271 INFO:teuthology.orchestra.run.smithi191.stdout:creating /etc/ceph/cluster2.client.mirror.keyring 2024-09-14T05:32:53.283 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster2.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.0.keyring 2024-09-14T05:32:53.364 INFO:teuthology.orchestra.run.smithi191.stdout:creating /etc/ceph/cluster2.client.mirror.0.keyring 2024-09-14T05:32:53.377 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster2.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.1.keyring 2024-09-14T05:32:53.458 INFO:teuthology.orchestra.run.smithi191.stdout:creating /etc/ceph/cluster2.client.mirror.1.keyring 2024-09-14T05:32:53.471 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster2.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.2.keyring 2024-09-14T05:32:53.551 INFO:teuthology.orchestra.run.smithi191.stdout:creating /etc/ceph/cluster2.client.mirror.2.keyring 2024-09-14T05:32:53.564 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster2.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.3.keyring 2024-09-14T05:32:53.644 INFO:teuthology.orchestra.run.smithi191.stdout:creating /etc/ceph/cluster2.client.mirror.3.keyring 2024-09-14T05:32:53.657 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster2.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.4.keyring 2024-09-14T05:32:53.737 INFO:teuthology.orchestra.run.smithi191.stdout:creating /etc/ceph/cluster2.client.mirror.4.keyring 2024-09-14T05:32:53.750 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster2.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.5.keyring 2024-09-14T05:32:53.831 INFO:teuthology.orchestra.run.smithi191.stdout:creating /etc/ceph/cluster2.client.mirror.5.keyring 2024-09-14T05:32:53.844 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster2.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.6.keyring 2024-09-14T05:32:53.926 INFO:teuthology.orchestra.run.smithi191.stdout:creating /etc/ceph/cluster2.client.mirror.6.keyring 2024-09-14T05:32:53.938 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-14T05:32:53.939 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi143.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi191.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-14T05:32:53.939 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/osd/cluster2-0 2024-09-14T05:32:54.003 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-14T05:32:54.003 INFO:tasks.ceph:role: cluster2.osd.0 2024-09-14T05:32:54.003 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi191.front.sepia.ceph.com 2024-09-14T05:32:54.003 DEBUG:teuthology.orchestra.run.smithi191:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-14T05:32:54.076 INFO:teuthology.orchestra.run.smithi191.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-14T05:32:54.076 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-14T05:32:54.076 INFO:teuthology.orchestra.run.smithi191.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-14T05:32:54.076 INFO:teuthology.orchestra.run.smithi191.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-14T05:32:54.076 INFO:teuthology.orchestra.run.smithi191.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-14T05:32:54.076 INFO:teuthology.orchestra.run.smithi191.stdout: = sunit=0 swidth=0 blks 2024-09-14T05:32:54.077 INFO:teuthology.orchestra.run.smithi191.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-14T05:32:54.077 INFO:teuthology.orchestra.run.smithi191.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-14T05:32:54.077 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-14T05:32:54.077 INFO:teuthology.orchestra.run.smithi191.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-14T05:32:54.581 INFO:teuthology.orchestra.run.smithi191.stdout:Discarding blocks...Done. 2024-09-14T05:32:54.583 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi191.front.sepia.ceph.com -o noatime 2024-09-14T05:32:54.583 DEBUG:teuthology.orchestra.run.smithi191:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster2-0 2024-09-14T05:32:54.709 DEBUG:teuthology.orchestra.run.smithi191:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-0 2024-09-14T05:32:54.741 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/osd/cluster2-1 2024-09-14T05:32:54.807 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-14T05:32:54.807 INFO:tasks.ceph:role: cluster2.osd.1 2024-09-14T05:32:54.807 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi191.front.sepia.ceph.com 2024-09-14T05:32:54.807 DEBUG:teuthology.orchestra.run.smithi191:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-14T05:32:54.882 INFO:teuthology.orchestra.run.smithi191.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-14T05:32:54.882 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-14T05:32:54.882 INFO:teuthology.orchestra.run.smithi191.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-14T05:32:54.882 INFO:teuthology.orchestra.run.smithi191.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-14T05:32:54.882 INFO:teuthology.orchestra.run.smithi191.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-14T05:32:54.882 INFO:teuthology.orchestra.run.smithi191.stdout: = sunit=0 swidth=0 blks 2024-09-14T05:32:54.882 INFO:teuthology.orchestra.run.smithi191.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-14T05:32:54.882 INFO:teuthology.orchestra.run.smithi191.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-14T05:32:54.882 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-14T05:32:54.882 INFO:teuthology.orchestra.run.smithi191.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-14T05:32:55.343 INFO:teuthology.orchestra.run.smithi191.stdout:Discarding blocks...Done. 2024-09-14T05:32:55.345 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi191.front.sepia.ceph.com -o noatime 2024-09-14T05:32:55.345 DEBUG:teuthology.orchestra.run.smithi191:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster2-1 2024-09-14T05:32:55.439 DEBUG:teuthology.orchestra.run.smithi191:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-1 2024-09-14T05:32:55.469 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/osd/cluster2-2 2024-09-14T05:32:55.535 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-14T05:32:55.535 INFO:tasks.ceph:role: cluster2.osd.2 2024-09-14T05:32:55.535 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi191.front.sepia.ceph.com 2024-09-14T05:32:55.536 DEBUG:teuthology.orchestra.run.smithi191:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-14T05:32:55.608 INFO:teuthology.orchestra.run.smithi191.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-14T05:32:55.608 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-14T05:32:55.608 INFO:teuthology.orchestra.run.smithi191.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-14T05:32:55.608 INFO:teuthology.orchestra.run.smithi191.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-14T05:32:55.608 INFO:teuthology.orchestra.run.smithi191.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-14T05:32:55.608 INFO:teuthology.orchestra.run.smithi191.stdout: = sunit=0 swidth=0 blks 2024-09-14T05:32:55.608 INFO:teuthology.orchestra.run.smithi191.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-14T05:32:55.609 INFO:teuthology.orchestra.run.smithi191.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-14T05:32:55.609 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-14T05:32:55.609 INFO:teuthology.orchestra.run.smithi191.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-14T05:32:56.100 INFO:teuthology.orchestra.run.smithi191.stdout:Discarding blocks...Done. 2024-09-14T05:32:56.102 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi191.front.sepia.ceph.com -o noatime 2024-09-14T05:32:56.102 DEBUG:teuthology.orchestra.run.smithi191:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster2-2 2024-09-14T05:32:56.251 DEBUG:teuthology.orchestra.run.smithi191:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-2 2024-09-14T05:32:56.283 DEBUG:teuthology.orchestra.run.smithi191:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-09-14T05:32:56.384 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-14T05:32:56.383+0000 7f41141db640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-0/keyring: can't open /var/lib/ceph/osd/cluster2-0/keyring: (2) No such file or directory 2024-09-14T05:32:56.388 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-14T05:32:56.387+0000 7f41141db640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-0/keyring 2024-09-14T05:32:56.388 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-14T05:32:56.387+0000 7f41141db640 -1 bdev(0x55e5c6271800 /var/lib/ceph/osd/cluster2-0/block) open stat got: (1) Operation not permitted 2024-09-14T05:32:56.388 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-14T05:32:56.387+0000 7f41141db640 -1 bluestore(/var/lib/ceph/osd/cluster2-0) _read_fsid unparsable uuid 2024-09-14T05:32:59.536 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-0 2024-09-14T05:32:59.566 DEBUG:teuthology.orchestra.run.smithi191:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-09-14T05:32:59.657 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-14T05:32:59.656+0000 7f413a444640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-1/keyring: can't open /var/lib/ceph/osd/cluster2-1/keyring: (2) No such file or directory 2024-09-14T05:32:59.657 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-14T05:32:59.656+0000 7f413a444640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-1/keyring 2024-09-14T05:32:59.657 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-14T05:32:59.656+0000 7f413a444640 -1 bdev(0x5594e3063800 /var/lib/ceph/osd/cluster2-1/block) open stat got: (1) Operation not permitted 2024-09-14T05:32:59.657 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-14T05:32:59.656+0000 7f413a444640 -1 bluestore(/var/lib/ceph/osd/cluster2-1) _read_fsid unparsable uuid 2024-09-14T05:33:03.061 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-1 2024-09-14T05:33:03.090 DEBUG:teuthology.orchestra.run.smithi191:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-09-14T05:33:03.181 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-14T05:33:03.180+0000 7fd52260d640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-2/keyring: can't open /var/lib/ceph/osd/cluster2-2/keyring: (2) No such file or directory 2024-09-14T05:33:03.181 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-14T05:33:03.180+0000 7fd52260d640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-2/keyring 2024-09-14T05:33:03.181 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-14T05:33:03.180+0000 7fd52260d640 -1 bdev(0x563188b23800 /var/lib/ceph/osd/cluster2-2/block) open stat got: (1) Operation not permitted 2024-09-14T05:33:03.182 INFO:teuthology.orchestra.run.smithi191.stderr:2024-09-14T05:33:03.180+0000 7fd52260d640 -1 bluestore(/var/lib/ceph/osd/cluster2-2) _read_fsid unparsable uuid 2024-09-14T05:33:06.109 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-2 2024-09-14T05:33:06.138 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-14T05:33:06.139 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-14T05:33:06.139 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/var/lib/ceph/mgr/cluster2-x/keyring of=/dev/stdout 2024-09-14T05:33:06.169 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-14T05:33:06.169 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/cluster2-0/keyring of=/dev/stdout 2024-09-14T05:33:06.207 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-14T05:33:06.207 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/cluster2-1/keyring of=/dev/stdout 2024-09-14T05:33:06.274 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-14T05:33:06.274 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/cluster2-2/keyring of=/dev/stdout 2024-09-14T05:33:06.341 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-09-14T05:33:06.341 DEBUG:teuthology.orchestra.run.smithi143:> dd if=/etc/ceph/cluster2.client.0.keyring of=/dev/stdout 2024-09-14T05:33:06.360 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-14T05:33:06.360 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/etc/ceph/cluster2.client.mirror.keyring of=/dev/stdout 2024-09-14T05:33:06.399 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-14T05:33:06.399 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/etc/ceph/cluster2.client.mirror.0.keyring of=/dev/stdout 2024-09-14T05:33:06.455 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-14T05:33:06.456 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/etc/ceph/cluster2.client.mirror.1.keyring of=/dev/stdout 2024-09-14T05:33:06.512 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-14T05:33:06.513 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/etc/ceph/cluster2.client.mirror.2.keyring of=/dev/stdout 2024-09-14T05:33:06.569 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-14T05:33:06.569 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/etc/ceph/cluster2.client.mirror.3.keyring of=/dev/stdout 2024-09-14T05:33:06.626 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-14T05:33:06.626 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/etc/ceph/cluster2.client.mirror.4.keyring of=/dev/stdout 2024-09-14T05:33:06.683 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-14T05:33:06.683 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/etc/ceph/cluster2.client.mirror.5.keyring of=/dev/stdout 2024-09-14T05:33:06.741 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-14T05:33:06.741 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/etc/ceph/cluster2.client.mirror.6.keyring of=/dev/stdout 2024-09-14T05:33:06.797 INFO:tasks.ceph:Adding keys to all mons... 2024-09-14T05:33:06.797 DEBUG:teuthology.orchestra.run.smithi191:> sudo tee -a /etc/ceph/cluster2.keyring 2024-09-14T05:33:06.862 INFO:teuthology.orchestra.run.smithi191.stdout:[mgr.x] 2024-09-14T05:33:06.862 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQAFIOVmf7haChAAo/+PBDjdCuYtVskkRJWrQg== 2024-09-14T05:33:06.862 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.0] 2024-09-14T05:33:06.862 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQAIIOVmY7jvFhAASBIYNp8Riph0x91YSIw7Kg== 2024-09-14T05:33:06.862 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.1] 2024-09-14T05:33:06.862 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQALIOVm9N8pJxAAQQq8Qqdew3CXvnNyaW7rBA== 2024-09-14T05:33:06.862 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.2] 2024-09-14T05:33:06.863 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQAPIOVmnRvMChAANLQYOwmRQTS5LAAUIu6LaQ== 2024-09-14T05:33:06.863 INFO:teuthology.orchestra.run.smithi191.stdout:[client.0] 2024-09-14T05:33:06.863 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQAFIOVmcyb1DBAAYH43CHRWMTR7Ho7jweanoQ== 2024-09-14T05:33:06.863 INFO:teuthology.orchestra.run.smithi191.stdout:[client.mirror] 2024-09-14T05:33:06.863 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQAFIOVmOhQrEBAAroTJE6Rn2QzWnQFcdxTItQ== 2024-09-14T05:33:06.863 INFO:teuthology.orchestra.run.smithi191.stdout:[client.mirror.0] 2024-09-14T05:33:06.863 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQAFIOVmP/jAFRAAkBOddDFBf5Inz0zyZd6MWw== 2024-09-14T05:33:06.863 INFO:teuthology.orchestra.run.smithi191.stdout:[client.mirror.1] 2024-09-14T05:33:06.863 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQAFIOVmOk5bGxAAPVmWRqojH7e5sUCUj3s6MQ== 2024-09-14T05:33:06.863 INFO:teuthology.orchestra.run.smithi191.stdout:[client.mirror.2] 2024-09-14T05:33:06.863 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQAFIOVmk+3hIBAACsIqp5L0yHMdNzgUIvz44g== 2024-09-14T05:33:06.863 INFO:teuthology.orchestra.run.smithi191.stdout:[client.mirror.3] 2024-09-14T05:33:06.863 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQAFIOVmBYlwJhAAXEPitfpAbMrzyoK539eX2g== 2024-09-14T05:33:06.864 INFO:teuthology.orchestra.run.smithi191.stdout:[client.mirror.4] 2024-09-14T05:33:06.864 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQAFIOVmTun4KxAAuxdA9u/g5IsnKQkwj8mVmA== 2024-09-14T05:33:06.864 INFO:teuthology.orchestra.run.smithi191.stdout:[client.mirror.5] 2024-09-14T05:33:06.864 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQAFIOVmKByYMRAAicySD7MU2e8yt2bIIAXfuQ== 2024-09-14T05:33:06.864 INFO:teuthology.orchestra.run.smithi191.stdout:[client.mirror.6] 2024-09-14T05:33:06.864 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQAFIOVmmNE4NxAAxxfD9CT8rZ/uCuv4aNao6Q== 2024-09-14T05:33:06.865 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-14T05:33:06.958 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-14T05:33:07.009 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-14T05:33:07.060 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-14T05:33:07.110 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-14T05:33:07.164 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-14T05:33:07.219 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-14T05:33:07.273 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-14T05:33:07.327 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-14T05:33:07.382 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-14T05:33:07.436 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-14T05:33:07.490 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-14T05:33:07.545 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-14T05:33:07.599 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-14T05:33:07.599 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/mon/cluster2-a 2024-09-14T05:33:07.623 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster2 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster2.monmap --keyring /etc/ceph/cluster2.keyring 2024-09-14T05:33:07.719 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster2-a 2024-09-14T05:33:07.743 DEBUG:teuthology.orchestra.run.smithi191:> rm -- /home/ubuntu/cephtest/cluster2.monmap 2024-09-14T05:33:07.798 INFO:tasks.ceph:Starting mon daemons in cluster cluster2... 2024-09-14T05:33:07.799 INFO:tasks.ceph.cluster2.mon.a:Restarting daemon 2024-09-14T05:33:07.799 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster2 -i a 2024-09-14T05:33:07.841 INFO:tasks.ceph.cluster2.mon.a:Started 2024-09-14T05:33:07.841 INFO:tasks.ceph:Starting mgr daemons in cluster cluster2... 2024-09-14T05:33:07.841 INFO:tasks.ceph.cluster2.mgr.x:Restarting daemon 2024-09-14T05:33:07.841 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster2 -i x 2024-09-14T05:33:07.844 INFO:tasks.ceph.cluster2.mgr.x:Started 2024-09-14T05:33:07.844 DEBUG:tasks.ceph:set 0 configs 2024-09-14T05:33:07.844 INFO:tasks.ceph:Setting crush tunables to default 2024-09-14T05:33:07.844 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster cluster2 osd crush tunables default 2024-09-14T05:33:08.014 INFO:teuthology.orchestra.run.smithi191.stderr:adjusted tunables profile to default 2024-09-14T05:33:08.025 INFO:tasks.ceph:check_enable_crimson: False 2024-09-14T05:33:08.025 INFO:tasks.ceph:Starting osd daemons in cluster cluster2... 2024-09-14T05:33:08.025 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-14T05:33:08.025 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/cluster2-0/fsid of=/dev/stdout 2024-09-14T05:33:08.050 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-14T05:33:08.050 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/cluster2-1/fsid of=/dev/stdout 2024-09-14T05:33:08.114 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-14T05:33:08.114 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/cluster2-2/fsid of=/dev/stdout 2024-09-14T05:33:08.179 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster cluster2 osd new 5275b373-5a00-4b1f-bbaf-b5a4d36f3d1a 0 2024-09-14T05:33:08.395 INFO:teuthology.orchestra.run.smithi191.stdout:0 2024-09-14T05:33:08.406 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster cluster2 osd new 74412469-f84e-4221-9d46-e16391bc87c8 1 2024-09-14T05:33:08.577 INFO:teuthology.orchestra.run.smithi191.stdout:1 2024-09-14T05:33:08.588 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster cluster2 osd new 015cffcb-af80-40d2-a86a-104cdb0167b8 2 2024-09-14T05:33:08.761 INFO:teuthology.orchestra.run.smithi191.stdout:2 2024-09-14T05:33:08.772 INFO:tasks.ceph.cluster2.osd.0:Restarting daemon 2024-09-14T05:33:08.772 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0 2024-09-14T05:33:08.774 INFO:tasks.ceph.cluster2.osd.0:Started 2024-09-14T05:33:08.774 INFO:tasks.ceph.cluster2.osd.1:Restarting daemon 2024-09-14T05:33:08.774 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 2024-09-14T05:33:08.776 INFO:tasks.ceph.cluster2.osd.1:Started 2024-09-14T05:33:08.776 INFO:tasks.ceph.cluster2.osd.2:Restarting daemon 2024-09-14T05:33:08.776 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 2024-09-14T05:33:08.778 INFO:tasks.ceph.cluster2.osd.2:Started 2024-09-14T05:33:08.779 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-14T05:33:08.949 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:33:08.949 INFO:teuthology.orchestra.run.smithi191.stdout:{"epoch":5,"fsid":"2fb493c9-6b9e-47da-a932-4884bef35060","created":"2024-09-14T05:33:07.928340+0000","modified":"2024-09-14T05:33:08.760360+0000","last_up_change":"0.000000","last_in_change":"2024-09-14T05:33:08.760360+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5275b373-5a00-4b1f-bbaf-b5a4d36f3d1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"74412469-f84e-4221-9d46-e16391bc87c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"015cffcb-af80-40d2-a86a-104cdb0167b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T05:33:08.959 INFO:tasks.ceph.ceph_manager.cluster2:[] 2024-09-14T05:33:08.960 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-14T05:33:09.262 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-09-14T05:33:09.423 INFO:teuthology.misc.health.smithi191.stdout: 2024-09-14T05:33:09.423 INFO:teuthology.misc.health.smithi191.stdout:{"epoch":5,"fsid":"2fb493c9-6b9e-47da-a932-4884bef35060","created":"2024-09-14T05:33:07.928340+0000","modified":"2024-09-14T05:33:08.760360+0000","last_up_change":"0.000000","last_in_change":"2024-09-14T05:33:08.760360+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5275b373-5a00-4b1f-bbaf-b5a4d36f3d1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"74412469-f84e-4221-9d46-e16391bc87c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"015cffcb-af80-40d2-a86a-104cdb0167b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T05:33:09.433 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-14T05:33:10.211 INFO:tasks.ceph.cluster2.osd.0.smithi191.stderr:2024-09-14T05:33:10.210+0000 7f75ed818640 -1 Falling back to public interface 2024-09-14T05:33:10.214 INFO:tasks.ceph.cluster2.osd.2.smithi191.stderr:2024-09-14T05:33:10.213+0000 7f3774018640 -1 Falling back to public interface 2024-09-14T05:33:10.450 INFO:tasks.ceph.cluster2.osd.1.smithi191.stderr:2024-09-14T05:33:10.450+0000 7f938680d640 -1 Falling back to public interface 2024-09-14T05:33:12.412 INFO:tasks.ceph.cluster2.mgr.x.smithi143.stderr:2024-09-14T05:33:12.411+0000 7f2459dfd640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-14T05:33:14.012 INFO:tasks.ceph.cluster2.osd.0.smithi191.stderr:2024-09-14T05:33:14.011+0000 7f75ed818640 -1 osd.0 0 log_to_monitors true 2024-09-14T05:33:14.245 INFO:tasks.ceph.cluster2.osd.2.smithi191.stderr:2024-09-14T05:33:14.244+0000 7f3774018640 -1 osd.2 0 log_to_monitors true 2024-09-14T05:33:14.456 INFO:tasks.ceph.cluster2.osd.1.smithi191.stderr:2024-09-14T05:33:14.455+0000 7f938680d640 -1 osd.1 0 log_to_monitors true 2024-09-14T05:33:15.736 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-09-14T05:33:16.127 INFO:teuthology.misc.health.smithi191.stdout: 2024-09-14T05:33:16.127 INFO:teuthology.misc.health.smithi191.stdout:{"epoch":7,"fsid":"2fb493c9-6b9e-47da-a932-4884bef35060","created":"2024-09-14T05:33:07.928340+0000","modified":"2024-09-14T05:33:15.414449+0000","last_up_change":"0.000000","last_in_change":"2024-09-14T05:33:08.760360+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5275b373-5a00-4b1f-bbaf-b5a4d36f3d1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"74412469-f84e-4221-9d46-e16391bc87c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"015cffcb-af80-40d2-a86a-104cdb0167b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T05:33:16.142 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-14T05:33:16.359 INFO:tasks.ceph.cluster2.osd.0.smithi191.stderr:2024-09-14T05:33:16.358+0000 7f75e909f640 -1 osd.0 0 waiting for initial osdmap 2024-09-14T05:33:16.366 INFO:tasks.ceph.cluster2.osd.0.smithi191.stderr:2024-09-14T05:33:16.366+0000 7f75e0ebe640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-14T05:33:16.371 INFO:tasks.ceph.cluster2.osd.2.smithi191.stderr:2024-09-14T05:33:16.370+0000 7f37701ae640 -1 osd.2 0 waiting for initial osdmap 2024-09-14T05:33:16.381 INFO:tasks.ceph.cluster2.osd.2.smithi191.stderr:2024-09-14T05:33:16.380+0000 7f37677b9640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-14T05:33:17.123 INFO:tasks.ceph.cluster2.osd.1.smithi191.stderr:2024-09-14T05:33:17.122+0000 7f9382932640 -1 osd.1 0 waiting for initial osdmap 2024-09-14T05:33:17.140 INFO:tasks.ceph.cluster2.osd.1.smithi191.stderr:2024-09-14T05:33:17.139+0000 7f9379f3d640 -1 osd.1 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-14T05:33:22.445 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-09-14T05:33:22.735 INFO:teuthology.misc.health.smithi191.stdout: 2024-09-14T05:33:22.735 INFO:teuthology.misc.health.smithi191.stdout:{"epoch":12,"fsid":"2fb493c9-6b9e-47da-a932-4884bef35060","created":"2024-09-14T05:33:07.928340+0000","modified":"2024-09-14T05:33:20.434414+0000","last_up_change":"2024-09-14T05:33:17.425068+0000","last_in_change":"2024-09-14T05:33:08.760360+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-14T05:33:17.570265+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5275b373-5a00-4b1f-bbaf-b5a4d36f3d1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":1406167874},{"type":"v1","addr":"172.21.15.191:6801","nonce":1406167874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":1406167874},{"type":"v1","addr":"172.21.15.191:6803","nonce":1406167874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":1406167874},{"type":"v1","addr":"172.21.15.191:6807","nonce":1406167874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":1406167874},{"type":"v1","addr":"172.21.15.191:6805","nonce":1406167874}]},"public_addr":"172.21.15.191:6801/1406167874","cluster_addr":"172.21.15.191:6803/1406167874","heartbeat_back_addr":"172.21.15.191:6807/1406167874","heartbeat_front_addr":"172.21.15.191:6805/1406167874","state":["exists","up"]},{"osd":1,"uuid":"74412469-f84e-4221-9d46-e16391bc87c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":805361069},{"type":"v1","addr":"172.21.15.191:6817","nonce":805361069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":805361069},{"type":"v1","addr":"172.21.15.191:6819","nonce":805361069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":805361069},{"type":"v1","addr":"172.21.15.191:6823","nonce":805361069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":805361069},{"type":"v1","addr":"172.21.15.191:6821","nonce":805361069}]},"public_addr":"172.21.15.191:6817/805361069","cluster_addr":"172.21.15.191:6819/805361069","heartbeat_back_addr":"172.21.15.191:6823/805361069","heartbeat_front_addr":"172.21.15.191:6821/805361069","state":["exists","up"]},{"osd":2,"uuid":"015cffcb-af80-40d2-a86a-104cdb0167b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":913831784},{"type":"v1","addr":"172.21.15.191:6809","nonce":913831784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":913831784},{"type":"v1","addr":"172.21.15.191:6811","nonce":913831784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":913831784},{"type":"v1","addr":"172.21.15.191:6815","nonce":913831784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":913831784},{"type":"v1","addr":"172.21.15.191:6813","nonce":913831784}]},"public_addr":"172.21.15.191:6809/913831784","cluster_addr":"172.21.15.191:6811/913831784","heartbeat_back_addr":"172.21.15.191:6815/913831784","heartbeat_front_addr":"172.21.15.191:6813/913831784","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T05:33:15.060141+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T05:33:15.465487+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T05:33:15.286411+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T05:33:22.746 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-14T05:33:22.746 INFO:tasks.ceph:Creating RBD pool 2024-09-14T05:33:22.747 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster cluster2 osd pool create rbd 8 2024-09-14T05:33:23.920 INFO:teuthology.orchestra.run.smithi191.stderr:pool 'rbd' created 2024-09-14T05:33:23.937 DEBUG:teuthology.orchestra.run.smithi191:> rbd --cluster cluster2 pool init rbd 2024-09-14T05:33:26.952 INFO:tasks.ceph:Starting mds daemons in cluster cluster2... 2024-09-14T05:33:26.953 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 config log 1 --format=json 2024-09-14T05:33:26.953 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-14T05:33:27.249 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:33:27.263 INFO:teuthology.orchestra.run.smithi191.stdout:[{"version":4,"timestamp":"2024-09-14T05:33:17.123964+0000","name":"","changes":[{"name":"osd.1/osd_mclock_max_capacity_iops_ssd","new_value":"7475.472861"}]}] 2024-09-14T05:33:27.263 INFO:tasks.ceph_manager:config epoch is 4 2024-09-14T05:33:27.264 INFO:tasks.ceph:Waiting until cluster2 daemons up and pgs clean... 2024-09-14T05:33:27.264 INFO:tasks.ceph.ceph_manager.cluster2:waiting for mgr available 2024-09-14T05:33:27.264 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 mgr dump --format=json 2024-09-14T05:33:27.595 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:33:27.606 INFO:teuthology.orchestra.run.smithi191.stdout:{"epoch":5,"flags":0,"active_gid":4098,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":190889430},{"type":"v1","addr":"172.21.15.143:6827","nonce":190889430}]},"active_addr":"172.21.15.143:6827/190889430","active_change":"2024-09-14T05:33:11.347714+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.143:0","nonce":4002443351}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.143:0","nonce":421982699}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.143:0","nonce":2942442597}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.143:0","nonce":428112682}]}]} 2024-09-14T05:33:27.609 INFO:tasks.ceph.ceph_manager.cluster2:mgr available! 2024-09-14T05:33:27.609 INFO:tasks.ceph.ceph_manager.cluster2:waiting for all up 2024-09-14T05:33:27.609 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-14T05:33:27.891 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:33:27.891 INFO:teuthology.orchestra.run.smithi191.stdout:{"epoch":16,"fsid":"2fb493c9-6b9e-47da-a932-4884bef35060","created":"2024-09-14T05:33:07.928340+0000","modified":"2024-09-14T05:33:26.930413+0000","last_up_change":"2024-09-14T05:33:17.425068+0000","last_in_change":"2024-09-14T05:33:08.760360+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-14T05:33:17.570265+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-14T05:33:23.035711+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5275b373-5a00-4b1f-bbaf-b5a4d36f3d1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":1406167874},{"type":"v1","addr":"172.21.15.191:6801","nonce":1406167874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":1406167874},{"type":"v1","addr":"172.21.15.191:6803","nonce":1406167874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":1406167874},{"type":"v1","addr":"172.21.15.191:6807","nonce":1406167874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":1406167874},{"type":"v1","addr":"172.21.15.191:6805","nonce":1406167874}]},"public_addr":"172.21.15.191:6801/1406167874","cluster_addr":"172.21.15.191:6803/1406167874","heartbeat_back_addr":"172.21.15.191:6807/1406167874","heartbeat_front_addr":"172.21.15.191:6805/1406167874","state":["exists","up"]},{"osd":1,"uuid":"74412469-f84e-4221-9d46-e16391bc87c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":805361069},{"type":"v1","addr":"172.21.15.191:6817","nonce":805361069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":805361069},{"type":"v1","addr":"172.21.15.191:6819","nonce":805361069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":805361069},{"type":"v1","addr":"172.21.15.191:6823","nonce":805361069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":805361069},{"type":"v1","addr":"172.21.15.191:6821","nonce":805361069}]},"public_addr":"172.21.15.191:6817/805361069","cluster_addr":"172.21.15.191:6819/805361069","heartbeat_back_addr":"172.21.15.191:6823/805361069","heartbeat_front_addr":"172.21.15.191:6821/805361069","state":["exists","up"]},{"osd":2,"uuid":"015cffcb-af80-40d2-a86a-104cdb0167b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":913831784},{"type":"v1","addr":"172.21.15.191:6809","nonce":913831784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":913831784},{"type":"v1","addr":"172.21.15.191:6811","nonce":913831784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":913831784},{"type":"v1","addr":"172.21.15.191:6815","nonce":913831784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":913831784},{"type":"v1","addr":"172.21.15.191:6813","nonce":913831784}]},"public_addr":"172.21.15.191:6809/913831784","cluster_addr":"172.21.15.191:6811/913831784","heartbeat_back_addr":"172.21.15.191:6815/913831784","heartbeat_front_addr":"172.21.15.191:6813/913831784","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T05:33:15.060141+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T05:33:15.465487+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T05:33:15.286411+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T05:33:27.904 INFO:tasks.ceph.ceph_manager.cluster2:all up! 2024-09-14T05:33:27.904 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-14T05:33:28.186 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:33:28.186 INFO:teuthology.orchestra.run.smithi191.stdout:{"epoch":16,"fsid":"2fb493c9-6b9e-47da-a932-4884bef35060","created":"2024-09-14T05:33:07.928340+0000","modified":"2024-09-14T05:33:26.930413+0000","last_up_change":"2024-09-14T05:33:17.425068+0000","last_in_change":"2024-09-14T05:33:08.760360+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-14T05:33:17.570265+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-14T05:33:23.035711+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5275b373-5a00-4b1f-bbaf-b5a4d36f3d1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":1406167874},{"type":"v1","addr":"172.21.15.191:6801","nonce":1406167874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":1406167874},{"type":"v1","addr":"172.21.15.191:6803","nonce":1406167874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":1406167874},{"type":"v1","addr":"172.21.15.191:6807","nonce":1406167874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":1406167874},{"type":"v1","addr":"172.21.15.191:6805","nonce":1406167874}]},"public_addr":"172.21.15.191:6801/1406167874","cluster_addr":"172.21.15.191:6803/1406167874","heartbeat_back_addr":"172.21.15.191:6807/1406167874","heartbeat_front_addr":"172.21.15.191:6805/1406167874","state":["exists","up"]},{"osd":1,"uuid":"74412469-f84e-4221-9d46-e16391bc87c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":805361069},{"type":"v1","addr":"172.21.15.191:6817","nonce":805361069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":805361069},{"type":"v1","addr":"172.21.15.191:6819","nonce":805361069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":805361069},{"type":"v1","addr":"172.21.15.191:6823","nonce":805361069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":805361069},{"type":"v1","addr":"172.21.15.191:6821","nonce":805361069}]},"public_addr":"172.21.15.191:6817/805361069","cluster_addr":"172.21.15.191:6819/805361069","heartbeat_back_addr":"172.21.15.191:6823/805361069","heartbeat_front_addr":"172.21.15.191:6821/805361069","state":["exists","up"]},{"osd":2,"uuid":"015cffcb-af80-40d2-a86a-104cdb0167b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":913831784},{"type":"v1","addr":"172.21.15.191:6809","nonce":913831784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":913831784},{"type":"v1","addr":"172.21.15.191:6811","nonce":913831784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":913831784},{"type":"v1","addr":"172.21.15.191:6815","nonce":913831784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":913831784},{"type":"v1","addr":"172.21.15.191:6813","nonce":913831784}]},"public_addr":"172.21.15.191:6809/913831784","cluster_addr":"172.21.15.191:6811/913831784","heartbeat_back_addr":"172.21.15.191:6815/913831784","heartbeat_front_addr":"172.21.15.191:6813/913831784","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T05:33:15.060141+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T05:33:15.465487+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T05:33:15.286411+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T05:33:28.199 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 flush_pg_stats 2024-09-14T05:33:28.200 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 flush_pg_stats 2024-09-14T05:33:28.200 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 flush_pg_stats 2024-09-14T05:33:28.340 INFO:teuthology.orchestra.run.smithi191.stdout:34359738373 2024-09-14T05:33:28.340 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-09-14T05:33:28.341 INFO:teuthology.orchestra.run.smithi191.stdout:34359738373 2024-09-14T05:33:28.342 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-09-14T05:33:28.343 INFO:teuthology.orchestra.run.smithi191.stdout:38654705669 2024-09-14T05:33:28.343 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-09-14T05:33:28.689 INFO:teuthology.orchestra.run.smithi191.stdout:34359738372 2024-09-14T05:33:28.693 INFO:teuthology.orchestra.run.smithi191.stdout:38654705668 2024-09-14T05:33:28.701 INFO:teuthology.orchestra.run.smithi191.stdout:34359738372 2024-09-14T05:33:28.703 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.0 2024-09-14T05:33:28.708 INFO:tasks.ceph.ceph_manager.cluster2:need seq 38654705669 got 38654705668 for osd.1 2024-09-14T05:33:28.711 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.2 2024-09-14T05:33:29.704 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-09-14T05:33:29.708 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-09-14T05:33:29.712 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-09-14T05:33:30.016 INFO:teuthology.orchestra.run.smithi191.stdout:34359738373 2024-09-14T05:33:30.030 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738373 for osd.0 2024-09-14T05:33:30.030 DEBUG:teuthology.parallel:result is None 2024-09-14T05:33:30.052 INFO:teuthology.orchestra.run.smithi191.stdout:34359738373 2024-09-14T05:33:30.060 INFO:teuthology.orchestra.run.smithi191.stdout:38654705670 2024-09-14T05:33:30.066 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738373 for osd.2 2024-09-14T05:33:30.066 DEBUG:teuthology.parallel:result is None 2024-09-14T05:33:30.072 INFO:tasks.ceph.ceph_manager.cluster2:need seq 38654705669 got 38654705670 for osd.1 2024-09-14T05:33:30.073 DEBUG:teuthology.parallel:result is None 2024-09-14T05:33:30.073 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-09-14T05:33:30.073 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-14T05:33:30.393 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:33:30.394 INFO:teuthology.orchestra.run.smithi191.stderr:dumped all 2024-09-14T05:33:30.405 INFO:teuthology.orchestra.run.smithi191.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-14T05:33:29.354496+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81604,"kb_used_data":520,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283033916,"statfs":{"total":289910292480,"available":289826729984,"internally_reserved":0,"allocated":532480,"data_stored":1300491,"data_compressed":5974,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.433573"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T05:33:26.938952+0000","last_change":"2024-09-14T05:33:26.938952+0000","last_active":"2024-09-14T05:33:26.938952+0000","last_peered":"2024-09-14T05:33:26.938952+0000","last_clean":"2024-09-14T05:33:26.938952+0000","last_became_active":"2024-09-14T05:33:24.930256+0000","last_became_peered":"2024-09-14T05:33:24.930256+0000","last_unstale":"2024-09-14T05:33:26.938952+0000","last_undegraded":"2024-09-14T05:33:26.938952+0000","last_fullsized":"2024-09-14T05:33:26.938952+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_clean_scrub_stamp":"2024-09-14T05:33:23.917964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:50:40.298612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087908799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T05:33:26.939287+0000","last_change":"2024-09-14T05:33:26.939287+0000","last_active":"2024-09-14T05:33:26.939287+0000","last_peered":"2024-09-14T05:33:26.939287+0000","last_clean":"2024-09-14T05:33:26.939287+0000","last_became_active":"2024-09-14T05:33:24.932185+0000","last_became_peered":"2024-09-14T05:33:24.932185+0000","last_unstale":"2024-09-14T05:33:26.939287+0000","last_undegraded":"2024-09-14T05:33:26.939287+0000","last_fullsized":"2024-09-14T05:33:26.939287+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_clean_scrub_stamp":"2024-09-14T05:33:23.917964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:46:00.286124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001112699,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T05:33:26.938327+0000","last_change":"2024-09-14T05:33:26.938327+0000","last_active":"2024-09-14T05:33:26.938327+0000","last_peered":"2024-09-14T05:33:26.938327+0000","last_clean":"2024-09-14T05:33:26.938327+0000","last_became_active":"2024-09-14T05:33:24.932193+0000","last_became_peered":"2024-09-14T05:33:24.932193+0000","last_unstale":"2024-09-14T05:33:26.938327+0000","last_undegraded":"2024-09-14T05:33:26.938327+0000","last_fullsized":"2024-09-14T05:33:26.938327+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_clean_scrub_stamp":"2024-09-14T05:33:23.917964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:39:48.398667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059268600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T05:33:26.938386+0000","last_change":"2024-09-14T05:33:26.938386+0000","last_active":"2024-09-14T05:33:26.938386+0000","last_peered":"2024-09-14T05:33:26.938386+0000","last_clean":"2024-09-14T05:33:26.938386+0000","last_became_active":"2024-09-14T05:33:24.931776+0000","last_became_peered":"2024-09-14T05:33:24.931776+0000","last_unstale":"2024-09-14T05:33:26.938386+0000","last_undegraded":"2024-09-14T05:33:26.938386+0000","last_fullsized":"2024-09-14T05:33:26.938386+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_clean_scrub_stamp":"2024-09-14T05:33:23.917964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:47:43.623946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061101000000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":22,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T05:33:26.935540+0000","last_change":"2024-09-14T05:33:26.935540+0000","last_active":"2024-09-14T05:33:26.935540+0000","last_peered":"2024-09-14T05:33:26.935540+0000","last_clean":"2024-09-14T05:33:26.935540+0000","last_became_active":"2024-09-14T05:33:24.931475+0000","last_became_peered":"2024-09-14T05:33:24.931475+0000","last_unstale":"2024-09-14T05:33:26.935540+0000","last_undegraded":"2024-09-14T05:33:26.935540+0000","last_fullsized":"2024-09-14T05:33:26.935540+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_clean_scrub_stamp":"2024-09-14T05:33:23.917964+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:22:40.175631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054839899999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T05:33:27.311986+0000","last_change":"2024-09-14T05:33:27.312188+0000","last_active":"2024-09-14T05:33:27.311986+0000","last_peered":"2024-09-14T05:33:27.311986+0000","last_clean":"2024-09-14T05:33:27.311986+0000","last_became_active":"2024-09-14T05:33:24.931144+0000","last_became_peered":"2024-09-14T05:33:24.931144+0000","last_unstale":"2024-09-14T05:33:27.311986+0000","last_undegraded":"2024-09-14T05:33:27.311986+0000","last_fullsized":"2024-09-14T05:33:27.311986+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_clean_scrub_stamp":"2024-09-14T05:33:23.917964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:00:35.641527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037866499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T05:33:27.311947+0000","last_change":"2024-09-14T05:33:27.312182+0000","last_active":"2024-09-14T05:33:27.311947+0000","last_peered":"2024-09-14T05:33:27.311947+0000","last_clean":"2024-09-14T05:33:27.311947+0000","last_became_active":"2024-09-14T05:33:24.931485+0000","last_became_peered":"2024-09-14T05:33:24.931485+0000","last_unstale":"2024-09-14T05:33:27.311947+0000","last_undegraded":"2024-09-14T05:33:27.311947+0000","last_fullsized":"2024-09-14T05:33:27.311947+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_clean_scrub_stamp":"2024-09-14T05:33:23.917964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:40:15.748455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039837999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":21,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T05:33:26.938911+0000","last_change":"2024-09-14T05:33:26.938911+0000","last_active":"2024-09-14T05:33:26.938911+0000","last_peered":"2024-09-14T05:33:26.938911+0000","last_clean":"2024-09-14T05:33:26.938911+0000","last_became_active":"2024-09-14T05:33:24.930770+0000","last_became_peered":"2024-09-14T05:33:24.930770+0000","last_unstale":"2024-09-14T05:33:26.938911+0000","last_undegraded":"2024-09-14T05:33:26.938911+0000","last_fullsized":"2024-09-14T05:33:26.938911+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_clean_scrub_stamp":"2024-09-14T05:33:23.917964+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:07:34.790624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010537719999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"11'52","reported_seq":88,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T05:33:26.938092+0000","last_change":"2024-09-14T05:33:19.761735+0000","last_active":"2024-09-14T05:33:26.938092+0000","last_peered":"2024-09-14T05:33:26.938092+0000","last_clean":"2024-09-14T05:33:26.938092+0000","last_became_active":"2024-09-14T05:33:19.760730+0000","last_became_peered":"2024-09-14T05:33:19.760730+0000","last_unstale":"2024-09-14T05:33:26.938092+0000","last_undegraded":"2024-09-14T05:33:26.938092+0000","last_fullsized":"2024-09-14T05:33:26.938092+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:33:18.427520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:33:18.427520+0000","last_clean_scrub_stamp":"2024-09-14T05:33:18.427520+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:04:05.440382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5974,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654705670,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":200704,"data_stored":631480,"data_compressed":2987,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":131072,"data_stored":37531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":200704,"data_stored":631480,"data_compressed":2987,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2987,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2987,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T05:33:30.406 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-14T05:33:30.688 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:33:30.688 INFO:teuthology.orchestra.run.smithi191.stderr:dumped all 2024-09-14T05:33:30.701 INFO:teuthology.orchestra.run.smithi191.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-14T05:33:29.354496+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81604,"kb_used_data":520,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283033916,"statfs":{"total":289910292480,"available":289826729984,"internally_reserved":0,"allocated":532480,"data_stored":1300491,"data_compressed":5974,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.433573"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T05:33:26.938952+0000","last_change":"2024-09-14T05:33:26.938952+0000","last_active":"2024-09-14T05:33:26.938952+0000","last_peered":"2024-09-14T05:33:26.938952+0000","last_clean":"2024-09-14T05:33:26.938952+0000","last_became_active":"2024-09-14T05:33:24.930256+0000","last_became_peered":"2024-09-14T05:33:24.930256+0000","last_unstale":"2024-09-14T05:33:26.938952+0000","last_undegraded":"2024-09-14T05:33:26.938952+0000","last_fullsized":"2024-09-14T05:33:26.938952+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_clean_scrub_stamp":"2024-09-14T05:33:23.917964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:50:40.298612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087908799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T05:33:26.939287+0000","last_change":"2024-09-14T05:33:26.939287+0000","last_active":"2024-09-14T05:33:26.939287+0000","last_peered":"2024-09-14T05:33:26.939287+0000","last_clean":"2024-09-14T05:33:26.939287+0000","last_became_active":"2024-09-14T05:33:24.932185+0000","last_became_peered":"2024-09-14T05:33:24.932185+0000","last_unstale":"2024-09-14T05:33:26.939287+0000","last_undegraded":"2024-09-14T05:33:26.939287+0000","last_fullsized":"2024-09-14T05:33:26.939287+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_clean_scrub_stamp":"2024-09-14T05:33:23.917964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:46:00.286124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001112699,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T05:33:26.938327+0000","last_change":"2024-09-14T05:33:26.938327+0000","last_active":"2024-09-14T05:33:26.938327+0000","last_peered":"2024-09-14T05:33:26.938327+0000","last_clean":"2024-09-14T05:33:26.938327+0000","last_became_active":"2024-09-14T05:33:24.932193+0000","last_became_peered":"2024-09-14T05:33:24.932193+0000","last_unstale":"2024-09-14T05:33:26.938327+0000","last_undegraded":"2024-09-14T05:33:26.938327+0000","last_fullsized":"2024-09-14T05:33:26.938327+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_clean_scrub_stamp":"2024-09-14T05:33:23.917964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:39:48.398667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059268600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T05:33:26.938386+0000","last_change":"2024-09-14T05:33:26.938386+0000","last_active":"2024-09-14T05:33:26.938386+0000","last_peered":"2024-09-14T05:33:26.938386+0000","last_clean":"2024-09-14T05:33:26.938386+0000","last_became_active":"2024-09-14T05:33:24.931776+0000","last_became_peered":"2024-09-14T05:33:24.931776+0000","last_unstale":"2024-09-14T05:33:26.938386+0000","last_undegraded":"2024-09-14T05:33:26.938386+0000","last_fullsized":"2024-09-14T05:33:26.938386+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_clean_scrub_stamp":"2024-09-14T05:33:23.917964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:47:43.623946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061101000000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":22,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T05:33:26.935540+0000","last_change":"2024-09-14T05:33:26.935540+0000","last_active":"2024-09-14T05:33:26.935540+0000","last_peered":"2024-09-14T05:33:26.935540+0000","last_clean":"2024-09-14T05:33:26.935540+0000","last_became_active":"2024-09-14T05:33:24.931475+0000","last_became_peered":"2024-09-14T05:33:24.931475+0000","last_unstale":"2024-09-14T05:33:26.935540+0000","last_undegraded":"2024-09-14T05:33:26.935540+0000","last_fullsized":"2024-09-14T05:33:26.935540+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_clean_scrub_stamp":"2024-09-14T05:33:23.917964+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:22:40.175631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054839899999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T05:33:27.311986+0000","last_change":"2024-09-14T05:33:27.312188+0000","last_active":"2024-09-14T05:33:27.311986+0000","last_peered":"2024-09-14T05:33:27.311986+0000","last_clean":"2024-09-14T05:33:27.311986+0000","last_became_active":"2024-09-14T05:33:24.931144+0000","last_became_peered":"2024-09-14T05:33:24.931144+0000","last_unstale":"2024-09-14T05:33:27.311986+0000","last_undegraded":"2024-09-14T05:33:27.311986+0000","last_fullsized":"2024-09-14T05:33:27.311986+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_clean_scrub_stamp":"2024-09-14T05:33:23.917964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:00:35.641527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037866499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T05:33:27.311947+0000","last_change":"2024-09-14T05:33:27.312182+0000","last_active":"2024-09-14T05:33:27.311947+0000","last_peered":"2024-09-14T05:33:27.311947+0000","last_clean":"2024-09-14T05:33:27.311947+0000","last_became_active":"2024-09-14T05:33:24.931485+0000","last_became_peered":"2024-09-14T05:33:24.931485+0000","last_unstale":"2024-09-14T05:33:27.311947+0000","last_undegraded":"2024-09-14T05:33:27.311947+0000","last_fullsized":"2024-09-14T05:33:27.311947+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_clean_scrub_stamp":"2024-09-14T05:33:23.917964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:40:15.748455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039837999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":21,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T05:33:26.938911+0000","last_change":"2024-09-14T05:33:26.938911+0000","last_active":"2024-09-14T05:33:26.938911+0000","last_peered":"2024-09-14T05:33:26.938911+0000","last_clean":"2024-09-14T05:33:26.938911+0000","last_became_active":"2024-09-14T05:33:24.930770+0000","last_became_peered":"2024-09-14T05:33:24.930770+0000","last_unstale":"2024-09-14T05:33:26.938911+0000","last_undegraded":"2024-09-14T05:33:26.938911+0000","last_fullsized":"2024-09-14T05:33:26.938911+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_clean_scrub_stamp":"2024-09-14T05:33:23.917964+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:07:34.790624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010537719999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"11'52","reported_seq":88,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-14T05:33:26.938092+0000","last_change":"2024-09-14T05:33:19.761735+0000","last_active":"2024-09-14T05:33:26.938092+0000","last_peered":"2024-09-14T05:33:26.938092+0000","last_clean":"2024-09-14T05:33:26.938092+0000","last_became_active":"2024-09-14T05:33:19.760730+0000","last_became_peered":"2024-09-14T05:33:19.760730+0000","last_unstale":"2024-09-14T05:33:26.938092+0000","last_undegraded":"2024-09-14T05:33:26.938092+0000","last_fullsized":"2024-09-14T05:33:26.938092+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:33:18.427520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:33:18.427520+0000","last_clean_scrub_stamp":"2024-09-14T05:33:18.427520+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:04:05.440382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5974,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654705670,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":200704,"data_stored":631480,"data_compressed":2987,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":131072,"data_stored":37531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":200704,"data_stored":631480,"data_compressed":2987,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2987,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2987,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T05:33:30.701 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-09-14T05:33:30.702 INFO:tasks.ceph:Waiting until ceph cluster cluster2 is healthy... 2024-09-14T05:33:30.702 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy 2024-09-14T05:33:30.702 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 health --format=json 2024-09-14T05:33:31.017 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T05:33:31.017 INFO:teuthology.orchestra.run.smithi191.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-14T05:33:31.030 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy done 2024-09-14T05:33:31.030 INFO:teuthology.run_tasks:Running task exec... 2024-09-14T05:33:31.506 INFO:teuthology.task.exec:Executing custom commands... 2024-09-14T05:33:31.506 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi191.front.sepia.ceph.com 2024-09-14T05:33:31.506 DEBUG:teuthology.orchestra.run.smithi191:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-09-14T05:33:31.871 INFO:teuthology.orchestra.run.smithi191.stdout:[client.mirror] 2024-09-14T05:33:31.872 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQDeH+Vmg2JuFhAASLB828bucYukAwh5vIqEFA== 2024-09-14T05:33:31.872 INFO:teuthology.orchestra.run.smithi191.stdout: caps mon = "profile rbd-mirror-peer" 2024-09-14T05:33:31.872 INFO:teuthology.orchestra.run.smithi191.stdout: caps osd = "profile rbd" 2024-09-14T05:33:31.872 INFO:teuthology.orchestra.run.smithi191.stderr:updated caps for client.mirror 2024-09-14T05:33:31.885 DEBUG:teuthology.orchestra.run.smithi191:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-14T05:33:32.250 INFO:teuthology.orchestra.run.smithi191.stdout:[client.mirror.0] 2024-09-14T05:33:32.250 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQDeH+VmywrXGRAAhXLA1PXPehaSfLmKqD4ddw== 2024-09-14T05:33:32.250 INFO:teuthology.orchestra.run.smithi191.stdout: caps mon = "profile rbd-mirror" 2024-09-14T05:33:32.251 INFO:teuthology.orchestra.run.smithi191.stdout: caps osd = "profile rbd" 2024-09-14T05:33:32.251 INFO:teuthology.orchestra.run.smithi191.stderr:updated caps for client.mirror.0 2024-09-14T05:33:32.264 DEBUG:teuthology.orchestra.run.smithi191:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-14T05:33:32.631 INFO:teuthology.orchestra.run.smithi191.stdout:[client.mirror.1] 2024-09-14T05:33:32.631 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQDeH+VmZPaQHxAAIjyukzX7dhPkNVTP4Q0bsg== 2024-09-14T05:33:32.631 INFO:teuthology.orchestra.run.smithi191.stdout: caps mon = "profile rbd-mirror" 2024-09-14T05:33:32.631 INFO:teuthology.orchestra.run.smithi191.stdout: caps osd = "profile rbd" 2024-09-14T05:33:32.631 INFO:teuthology.orchestra.run.smithi191.stderr:updated caps for client.mirror.1 2024-09-14T05:33:32.645 DEBUG:teuthology.orchestra.run.smithi191:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-14T05:33:33.007 INFO:teuthology.orchestra.run.smithi191.stdout:[client.mirror.2] 2024-09-14T05:33:33.008 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQDeH+Vmngc4JRAAceWUCBxyDe0DK2ZJsSAkwA== 2024-09-14T05:33:33.008 INFO:teuthology.orchestra.run.smithi191.stdout: caps mon = "profile rbd-mirror" 2024-09-14T05:33:33.008 INFO:teuthology.orchestra.run.smithi191.stdout: caps osd = "profile rbd" 2024-09-14T05:33:33.008 INFO:teuthology.orchestra.run.smithi191.stderr:updated caps for client.mirror.2 2024-09-14T05:33:33.021 DEBUG:teuthology.orchestra.run.smithi191:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-14T05:33:33.386 INFO:teuthology.orchestra.run.smithi191.stdout:[client.mirror.3] 2024-09-14T05:33:33.386 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQDeH+VmGxjWKhAAVGh9ekAahsA+/81mhUOZlA== 2024-09-14T05:33:33.386 INFO:teuthology.orchestra.run.smithi191.stdout: caps mon = "profile rbd-mirror" 2024-09-14T05:33:33.386 INFO:teuthology.orchestra.run.smithi191.stdout: caps osd = "profile rbd" 2024-09-14T05:33:33.387 INFO:teuthology.orchestra.run.smithi191.stderr:updated caps for client.mirror.3 2024-09-14T05:33:33.400 DEBUG:teuthology.orchestra.run.smithi191:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-09-14T05:33:33.762 INFO:teuthology.orchestra.run.smithi191.stdout:[client.mirror] 2024-09-14T05:33:33.763 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQAFIOVmOhQrEBAAroTJE6Rn2QzWnQFcdxTItQ== 2024-09-14T05:33:33.763 INFO:teuthology.orchestra.run.smithi191.stdout: caps mon = "profile rbd-mirror-peer" 2024-09-14T05:33:33.763 INFO:teuthology.orchestra.run.smithi191.stdout: caps osd = "profile rbd" 2024-09-14T05:33:33.763 INFO:teuthology.orchestra.run.smithi191.stderr:updated caps for client.mirror 2024-09-14T05:33:33.777 DEBUG:teuthology.orchestra.run.smithi191:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-14T05:33:34.145 INFO:teuthology.orchestra.run.smithi191.stdout:[client.mirror.0] 2024-09-14T05:33:34.145 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQAFIOVmP/jAFRAAkBOddDFBf5Inz0zyZd6MWw== 2024-09-14T05:33:34.145 INFO:teuthology.orchestra.run.smithi191.stdout: caps mon = "profile rbd-mirror" 2024-09-14T05:33:34.145 INFO:teuthology.orchestra.run.smithi191.stdout: caps osd = "profile rbd" 2024-09-14T05:33:34.146 INFO:teuthology.orchestra.run.smithi191.stderr:updated caps for client.mirror.0 2024-09-14T05:33:34.159 DEBUG:teuthology.orchestra.run.smithi191:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-14T05:33:34.520 INFO:teuthology.orchestra.run.smithi191.stdout:[client.mirror.1] 2024-09-14T05:33:34.521 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQAFIOVmOk5bGxAAPVmWRqojH7e5sUCUj3s6MQ== 2024-09-14T05:33:34.521 INFO:teuthology.orchestra.run.smithi191.stdout: caps mon = "profile rbd-mirror" 2024-09-14T05:33:34.521 INFO:teuthology.orchestra.run.smithi191.stdout: caps osd = "profile rbd" 2024-09-14T05:33:34.521 INFO:teuthology.orchestra.run.smithi191.stderr:updated caps for client.mirror.1 2024-09-14T05:33:34.535 DEBUG:teuthology.orchestra.run.smithi191:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-14T05:33:34.902 INFO:teuthology.orchestra.run.smithi191.stdout:[client.mirror.2] 2024-09-14T05:33:34.902 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQAFIOVmk+3hIBAACsIqp5L0yHMdNzgUIvz44g== 2024-09-14T05:33:34.902 INFO:teuthology.orchestra.run.smithi191.stdout: caps mon = "profile rbd-mirror" 2024-09-14T05:33:34.902 INFO:teuthology.orchestra.run.smithi191.stdout: caps osd = "profile rbd" 2024-09-14T05:33:34.903 INFO:teuthology.orchestra.run.smithi191.stderr:updated caps for client.mirror.2 2024-09-14T05:33:34.916 DEBUG:teuthology.orchestra.run.smithi191:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-14T05:33:35.283 INFO:teuthology.orchestra.run.smithi191.stdout:[client.mirror.3] 2024-09-14T05:33:35.283 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQAFIOVmBYlwJhAAXEPitfpAbMrzyoK539eX2g== 2024-09-14T05:33:35.283 INFO:teuthology.orchestra.run.smithi191.stdout: caps mon = "profile rbd-mirror" 2024-09-14T05:33:35.283 INFO:teuthology.orchestra.run.smithi191.stdout: caps osd = "profile rbd" 2024-09-14T05:33:35.284 INFO:teuthology.orchestra.run.smithi191.stderr:updated caps for client.mirror.3 2024-09-14T05:33:35.297 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-14T05:33:35.307 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-14T05:33:35.307 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-14T05:33:35.310 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-14T05:33:35.310 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-14T05:33:35.318 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-14T05:33:35.318 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-14T05:33:35.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-14T05:33:35.321 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-14T05:33:35.331 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-09-14T05:33:35.331 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-09-14T05:33:35.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-09-14T05:33:35.333 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-14T05:33:35.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-09-14T05:33:35.342 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-09-14T05:33:35.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-09-14T05:33:35.344 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-14T05:33:35.353 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-14T05:33:35.354 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-14T05:33:35.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-14T05:33:35.356 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-14T05:33:35.366 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-14T05:33:35.366 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-14T05:33:35.368 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-14T05:33:35.368 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-14T05:33:35.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-14T05:33:35.377 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-14T05:33:35.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-14T05:33:35.380 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-14T05:33:35.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-09-14T05:33:35.389 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-09-14T05:33:35.392 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-09-14T05:33:35.392 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2024-09-14T05:33:35.402 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1726292015 2024-09-14T05:33:35.402 DEBUG:tasks.rbd_mirror_thrash:Yielding 2024-09-14T05:33:35.403 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2024-09-14T05:33:35.413 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1726292015 2024-09-14T05:33:35.413 DEBUG:tasks.rbd_mirror_thrash:Yielding 2024-09-14T05:33:35.413 INFO:teuthology.run_tasks:Running task workunit... 2024-09-14T05:33:35.433 INFO:tasks.workunit:Pulling workunits from ref 0d61375b8ab53be4c233f5b93e463edfdff9316a 2024-09-14T05:33:35.434 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-14T05:33:35.434 DEBUG:teuthology.orchestra.run.smithi191:> stat -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-14T05:33:35.434 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster1 2024-09-14T05:33:35.434 INFO:tasks.rbd_mirror_thrash:waiting for 111 secs before thrashing 2024-09-14T05:33:35.435 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster2 2024-09-14T05:33:35.435 INFO:tasks.rbd_mirror_thrash:waiting for 78 secs before thrashing 2024-09-14T05:33:35.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T05:33:35.455 INFO:teuthology.orchestra.run.smithi191.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.cluster1.mirror': No such file or directory 2024-09-14T05:33:35.455 DEBUG:teuthology.orchestra.run.smithi191:> mkdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-14T05:33:35.511 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-14T05:33:35.511 DEBUG:teuthology.orchestra.run.smithi191:> cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror && mkdir -- client.mirror 2024-09-14T05:33:35.565 INFO:tasks.workunit:timeout=6h 2024-09-14T05:33:35.565 INFO:tasks.workunit:cleanup=True 2024-09-14T05:33:35.566 DEBUG:teuthology.orchestra.run.smithi191:> rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout 0d61375b8ab53be4c233f5b93e463edfdff9316a 2024-09-14T05:33:35.621 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Cloning into '/home/ubuntu/cephtest/clone.cluster1.client.mirror'... 2024-09-14T05:34:32.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Note: switching to '0d61375b8ab53be4c233f5b93e463edfdff9316a'. 2024-09-14T05:34:32.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:34:32.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-14T05:34:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-14T05:34:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:state without impacting any branches by switching back to a branch. 2024-09-14T05:34:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:34:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-14T05:34:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-14T05:34:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:34:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: git switch -c 2024-09-14T05:34:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:34:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Or undo this operation with: 2024-09-14T05:34:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:34:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: git switch - 2024-09-14T05:34:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:34:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-14T05:34:32.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:34:32.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:HEAD is now at 0d61375b8ab Merge pull request #59485 from mcv21/cephadm_image_warn 2024-09-14T05:34:32.125 DEBUG:teuthology.orchestra.run.smithi191:> cd -- /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror 2024-09-14T05:34:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-14T05:34:32.220 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-09-14T05:34:32.220 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-14T05:34:33.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-14T05:34:33.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-14T05:34:33.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-09-14T05:34:33.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-09-14T05:34:33.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-14T05:34:33.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-09-14T05:34:33.151 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-14T05:34:33.151 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror of=/dev/stdout 2024-09-14T05:34:33.167 INFO:tasks.workunit:Running workunits matching rbd/rbd_mirror_stress.sh on cluster1.client.mirror... 2024-09-14T05:34:33.168 INFO:tasks.workunit:Running workunit rbd/rbd_mirror_stress.sh... 2024-09-14T05:34:33.168 DEBUG:teuthology.orchestra.run.smithi191:workunit test rbd/rbd_mirror_stress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0d61375b8ab53be4c233f5b93e463edfdff9316a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS='' MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_INSTANCES=4 RBD_MIRROR_MODE=snapshot RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-09-14T05:34:33.226 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ IMAGE_COUNT=50 2024-09-14T05:34:33.226 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ export LOCKDEP=0 2024-09-14T05:34:33.226 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ LOCKDEP=0 2024-09-14T05:34:33.227 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-09-14T05:34:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ . /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_helpers.sh 2024-09-14T05:34:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ RBD_MIRROR_INSTANCES=4 2024-09-14T05:34:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CLUSTER1=cluster1 2024-09-14T05:34:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CLUSTER2=cluster2 2024-09-14T05:34:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ PEER_CLUSTER_SUFFIX= 2024-09-14T05:34:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ POOL=mirror 2024-09-14T05:34:33.229 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ PARENT_POOL=mirror_parent 2024-09-14T05:34:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ NS1=ns1 2024-09-14T05:34:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ NS2=ns2 2024-09-14T05:34:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ TEMPDIR= 2024-09-14T05:34:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CEPH_ID=mirror 2024-09-14T05:34:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff 2024-09-14T05:34:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ MIRROR_USER_ID_PREFIX=mirror. 2024-09-14T05:34:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ RBD_MIRROR_MODE=snapshot 2024-09-14T05:34:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ MIRROR_POOL_MODE=image 2024-09-14T05:34:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' snapshot = snapshot ']' 2024-09-14T05:34:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ MIRROR_POOL_MODE=image 2024-09-14T05:34:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ export 'CEPH_ARGS=--id mirror' 2024-09-14T05:34:33.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CEPH_ARGS='--id mirror' 2024-09-14T05:34:33.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ LAST_MIRROR_INSTANCE=3 2024-09-14T05:34:33.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-09-14T05:34:33.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ readlink -f /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/../../../src 2024-09-14T05:34:33.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror/src 2024-09-14T05:34:33.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CEPH_BIN=. 2024-09-14T05:34:33.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CEPH_SRC=. 2024-09-14T05:34:33.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' -e CMakeCache.txt ']' 2024-09-14T05:34:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 -gt 0 ']' 2024-09-14T05:34:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ setup 2024-09-14T05:34:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local c 2024-09-14T05:34:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ trap 'cleanup $?' INT TERM EXIT 2024-09-14T05:34:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ setup_tempdir 2024-09-14T05:34:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:34:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mktemp -d 2024-09-14T05:34:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ TEMPDIR=/tmp/tmp.sXArEbIrwv 2024-09-14T05:34:33.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -z 1 ']' 2024-09-14T05:34:33.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ setup_pools cluster1 cluster2 2024-09-14T05:34:33.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:34:33.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local remote_cluster=cluster2 2024-09-14T05:34:33.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mon_map_file 2024-09-14T05:34:33.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mon_addr 2024-09-14T05:34:33.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local admin_key_file 2024-09-14T05:34:33.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local uuid 2024-09-14T05:34:33.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ CEPH_ARGS= 2024-09-14T05:34:33.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ ceph --cluster cluster1 osd pool create mirror 64 64 2024-09-14T05:34:33.951 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:pool 'mirror' already exists 2024-09-14T05:34:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ CEPH_ARGS= 2024-09-14T05:34:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ ceph --cluster cluster1 osd pool create mirror_parent 64 64 2024-09-14T05:34:34.952 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:pool 'mirror_parent' already exists 2024-09-14T05:34:34.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ CEPH_ARGS= 2024-09-14T05:34:34.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 pool init mirror 2024-09-14T05:34:37.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ CEPH_ARGS= 2024-09-14T05:34:37.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 pool init mirror_parent 2024-09-14T05:34:40.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:34:40.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ CEPH_ARGS= 2024-09-14T05:34:40.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 mirror pool enable --site-name cluster1 mirror image 2024-09-14T05:34:40.952 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent image 2024-09-14T05:34:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 namespace create mirror/ns1 2024-09-14T05:34:41.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 namespace create mirror/ns2 2024-09-14T05:34:41.080 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns1 image 2024-09-14T05:34:41.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns2 image 2024-09-14T05:34:41.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -z ']' 2024-09-14T05:34:41.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -z ']' 2024-09-14T05:34:41.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ peer_add cluster1 mirror cluster2 2024-09-14T05:34:41.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:34:41.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:34:41.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:34:41.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:34:41.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local client_cluster=cluster2 2024-09-14T05:34:41.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:34:41.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local remote_cluster=cluster2 2024-09-14T05:34:41.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local uuid_var_name 2024-09-14T05:34:41.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:34:41.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local error_code 2024-09-14T05:34:41.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local peer_uuid 2024-09-14T05:34:41.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 16 32 2024-09-14T05:34:41.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:34:41.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror cluster2 2024-09-14T05:34:41.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ peer_uuid=e080edb4-e541-436f-9ab3-27e6d1d7a589 2024-09-14T05:34:41.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=0 2024-09-14T05:34:41.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:34:41.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 0 -eq 17 ']' 2024-09-14T05:34:41.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 0 -eq 0 2024-09-14T05:34:41.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:34:41.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:34:41.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ peer_add cluster1 mirror_parent cluster2 2024-09-14T05:34:41.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:34:41.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:34:41.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror_parent 2024-09-14T05:34:41.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:34:41.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local client_cluster=cluster2 2024-09-14T05:34:41.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:34:41.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local remote_cluster=cluster2 2024-09-14T05:34:41.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local uuid_var_name 2024-09-14T05:34:41.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:34:41.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local error_code 2024-09-14T05:34:41.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local peer_uuid 2024-09-14T05:34:41.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 16 32 2024-09-14T05:34:41.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:34:41.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror_parent cluster2 2024-09-14T05:34:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ peer_uuid=405507fe-2e20-40ff-b8dc-18bbb926dfc5 2024-09-14T05:34:41.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=0 2024-09-14T05:34:41.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:34:41.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 0 -eq 17 ']' 2024-09-14T05:34:41.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 0 -eq 0 2024-09-14T05:34:41.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:34:41.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:34:41.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ setup_pools cluster2 cluster1 2024-09-14T05:34:41.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:34:41.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local remote_cluster=cluster1 2024-09-14T05:34:41.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mon_map_file 2024-09-14T05:34:41.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mon_addr 2024-09-14T05:34:41.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local admin_key_file 2024-09-14T05:34:41.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local uuid 2024-09-14T05:34:41.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ CEPH_ARGS= 2024-09-14T05:34:41.250 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ ceph --cluster cluster2 osd pool create mirror 64 64 2024-09-14T05:34:42.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:pool 'mirror' already exists 2024-09-14T05:34:42.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ CEPH_ARGS= 2024-09-14T05:34:42.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ ceph --cluster cluster2 osd pool create mirror_parent 64 64 2024-09-14T05:34:43.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:pool 'mirror_parent' already exists 2024-09-14T05:34:43.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ CEPH_ARGS= 2024-09-14T05:34:43.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 pool init mirror 2024-09-14T05:34:46.478 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ CEPH_ARGS= 2024-09-14T05:34:46.478 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 pool init mirror_parent 2024-09-14T05:34:49.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:34:49.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ CEPH_ARGS= 2024-09-14T05:34:49.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror pool enable --site-name cluster2 mirror image 2024-09-14T05:34:49.546 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent image 2024-09-14T05:34:49.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 namespace create mirror/ns1 2024-09-14T05:34:49.631 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 namespace create mirror/ns2 2024-09-14T05:34:49.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns1 image 2024-09-14T05:34:49.710 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns2 image 2024-09-14T05:34:49.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -z ']' 2024-09-14T05:34:49.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -z ']' 2024-09-14T05:34:49.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ peer_add cluster2 mirror cluster1 2024-09-14T05:34:49.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:34:49.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:34:49.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:34:49.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:34:49.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local client_cluster=cluster1 2024-09-14T05:34:49.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:34:49.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local remote_cluster=cluster1 2024-09-14T05:34:49.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local uuid_var_name 2024-09-14T05:34:49.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:34:49.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local error_code 2024-09-14T05:34:49.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local peer_uuid 2024-09-14T05:34:49.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 16 32 2024-09-14T05:34:49.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:34:49.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror cluster1 2024-09-14T05:34:49.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ peer_uuid=cb2fb98b-e651-4f26-a0cf-f42eab4c0247 2024-09-14T05:34:49.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=0 2024-09-14T05:34:49.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:34:49.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 0 -eq 17 ']' 2024-09-14T05:34:49.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 0 -eq 0 2024-09-14T05:34:49.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:34:49.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:34:49.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ peer_add cluster2 mirror_parent cluster1 2024-09-14T05:34:49.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:34:49.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:34:49.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror_parent 2024-09-14T05:34:49.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:34:49.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local client_cluster=cluster1 2024-09-14T05:34:49.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:34:49.794 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local remote_cluster=cluster1 2024-09-14T05:34:49.794 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local uuid_var_name 2024-09-14T05:34:49.794 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:34:49.794 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local error_code 2024-09-14T05:34:49.794 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local peer_uuid 2024-09-14T05:34:49.794 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 16 32 2024-09-14T05:34:49.794 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:34:49.794 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror_parent cluster1 2024-09-14T05:34:49.832 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ peer_uuid=915218d2-3a21-4d40-8ef2-cbc55f23bf26 2024-09-14T05:34:49.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=0 2024-09-14T05:34:49.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:34:49.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 0 -eq 17 ']' 2024-09-14T05:34:49.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 0 -eq 0 2024-09-14T05:34:49.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:34:49.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:34:49.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:34:49.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ start_mirrors cluster1 2024-09-14T05:34:49.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:34:49.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ seq 0 3 2024-09-14T05:34:49.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:34:49.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ start_mirror cluster1:0 2024-09-14T05:34:49.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1:0 2024-09-14T05:34:49.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local instance 2024-09-14T05:34:49.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:34:49.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local val=cluster1:0 2024-09-14T05:34:49.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster_var_name=cluster 2024-09-14T05:34:49.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local instance_var_name=instance 2024-09-14T05:34:49.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cluster=cluster1 2024-09-14T05:34:49.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ instance=0 2024-09-14T05:34:49.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-14T05:34:49.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ eval cluster=cluster1 2024-09-14T05:34:49.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:34:49.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ eval instance=0 2024-09-14T05:34:49.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:34:49.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 1 2024-09-14T05:34:49.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 2024-09-14T05:34:49.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:34:49.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ start_mirror cluster1:1 2024-09-14T05:34:49.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1:1 2024-09-14T05:34:49.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local instance 2024-09-14T05:34:49.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-14T05:34:49.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local val=cluster1:1 2024-09-14T05:34:49.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster_var_name=cluster 2024-09-14T05:34:49.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local instance_var_name=instance 2024-09-14T05:34:49.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cluster=cluster1 2024-09-14T05:34:49.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ instance=1 2024-09-14T05:34:49.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-14T05:34:49.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ eval cluster=cluster1 2024-09-14T05:34:49.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:34:49.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ eval instance=1 2024-09-14T05:34:49.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-14T05:34:49.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 1 2024-09-14T05:34:49.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 2024-09-14T05:34:49.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:34:49.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ start_mirror cluster1:2 2024-09-14T05:34:49.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1:2 2024-09-14T05:34:49.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local instance 2024-09-14T05:34:49.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-09-14T05:34:49.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local val=cluster1:2 2024-09-14T05:34:49.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster_var_name=cluster 2024-09-14T05:34:49.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local instance_var_name=instance 2024-09-14T05:34:49.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cluster=cluster1 2024-09-14T05:34:49.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ instance=2 2024-09-14T05:34:49.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 2 = cluster1:2 ']' 2024-09-14T05:34:49.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ eval cluster=cluster1 2024-09-14T05:34:49.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:34:49.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ eval instance=2 2024-09-14T05:34:49.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-14T05:34:49.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 1 2024-09-14T05:34:49.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 2024-09-14T05:34:49.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:34:49.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ start_mirror cluster1:3 2024-09-14T05:34:49.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1:3 2024-09-14T05:34:49.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local instance 2024-09-14T05:34:49.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-09-14T05:34:49.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local val=cluster1:3 2024-09-14T05:34:49.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster_var_name=cluster 2024-09-14T05:34:49.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local instance_var_name=instance 2024-09-14T05:34:49.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cluster=cluster1 2024-09-14T05:34:49.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ instance=3 2024-09-14T05:34:49.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 3 = cluster1:3 ']' 2024-09-14T05:34:49.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ eval cluster=cluster1 2024-09-14T05:34:49.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:34:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ eval instance=3 2024-09-14T05:34:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=3 2024-09-14T05:34:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 1 2024-09-14T05:34:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 2024-09-14T05:34:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ start_mirrors cluster2 2024-09-14T05:34:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:34:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ seq 0 3 2024-09-14T05:34:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:34:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ start_mirror cluster2:0 2024-09-14T05:34:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2:0 2024-09-14T05:34:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local instance 2024-09-14T05:34:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-14T05:34:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local val=cluster2:0 2024-09-14T05:34:49.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster_var_name=cluster 2024-09-14T05:34:49.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local instance_var_name=instance 2024-09-14T05:34:49.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cluster=cluster2 2024-09-14T05:34:49.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ instance=0 2024-09-14T05:34:49.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-14T05:34:49.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ eval cluster=cluster2 2024-09-14T05:34:49.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster2 2024-09-14T05:34:49.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ eval instance=0 2024-09-14T05:34:49.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:34:49.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 1 2024-09-14T05:34:49.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 2024-09-14T05:34:49.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:34:49.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ start_mirror cluster2:1 2024-09-14T05:34:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2:1 2024-09-14T05:34:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local instance 2024-09-14T05:34:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-09-14T05:34:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local val=cluster2:1 2024-09-14T05:34:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster_var_name=cluster 2024-09-14T05:34:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local instance_var_name=instance 2024-09-14T05:34:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cluster=cluster2 2024-09-14T05:34:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ instance=1 2024-09-14T05:34:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 1 = cluster2:1 ']' 2024-09-14T05:34:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ eval cluster=cluster2 2024-09-14T05:34:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster2 2024-09-14T05:34:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ eval instance=1 2024-09-14T05:34:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-14T05:34:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 1 2024-09-14T05:34:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 2024-09-14T05:34:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:34:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ start_mirror cluster2:2 2024-09-14T05:34:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2:2 2024-09-14T05:34:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local instance 2024-09-14T05:34:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set_cluster_instance cluster2:2 cluster instance 2024-09-14T05:34:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local val=cluster2:2 2024-09-14T05:34:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster_var_name=cluster 2024-09-14T05:34:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local instance_var_name=instance 2024-09-14T05:34:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cluster=cluster2 2024-09-14T05:34:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ instance=2 2024-09-14T05:34:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 2 = cluster2:2 ']' 2024-09-14T05:34:49.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ eval cluster=cluster2 2024-09-14T05:34:49.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster2 2024-09-14T05:34:49.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ eval instance=2 2024-09-14T05:34:49.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-14T05:34:49.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 1 2024-09-14T05:34:49.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 2024-09-14T05:34:49.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:34:49.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ start_mirror cluster2:3 2024-09-14T05:34:49.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2:3 2024-09-14T05:34:49.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local instance 2024-09-14T05:34:49.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set_cluster_instance cluster2:3 cluster instance 2024-09-14T05:34:49.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local val=cluster2:3 2024-09-14T05:34:49.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster_var_name=cluster 2024-09-14T05:34:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local instance_var_name=instance 2024-09-14T05:34:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cluster=cluster2 2024-09-14T05:34:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ instance=3 2024-09-14T05:34:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 3 = cluster2:3 ']' 2024-09-14T05:34:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ eval cluster=cluster2 2024-09-14T05:34:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster2 2024-09-14T05:34:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ eval instance=3 2024-09-14T05:34:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=3 2024-09-14T05:34:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 1 2024-09-14T05:34:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 2024-09-14T05:34:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ testlog 'TEST: add image and test replay after client crashes' 2024-09-14T05:34:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ tee -a /tmp/tmp.sXArEbIrwv/rbd-mirror.test.log 2024-09-14T05:34:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ date '+%F %T' 2024-09-14T05:34:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo 2024-09-14 05:34:49 TEST: add image and test replay after client crashes 2024-09-14T05:34:49.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:2024-09-14 05:34:49 TEST: add image and test replay after client crashes 2024-09-14T05:34:49.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=test 2024-09-14T05:34:49.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror test snapshot 512M 2024-09-14T05:34:49.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:34:49.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:34:49.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:34:49.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:34:49.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:34:49.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:34:49.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:34:49.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-14T05:34:49.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:34:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror test 512M 2024-09-14T05:34:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:34:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:34:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:34:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:34:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:34:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:34:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T05:34:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 512M ']' 2024-09-14T05:34:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=512M 2024-09-14T05:34:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:34:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/test 2024-09-14T05:34:49.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T05:34:49.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:34:49.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-14T05:34:49.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:34:49.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror test snapshot 2024-09-14T05:34:49.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:34:49.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:34:49.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:34:49.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:34:49.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/test snapshot 2024-09-14T05:34:50.556 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T05:34:50.563 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/test 2024-09-14T05:34:50.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'test': 2024-09-14T05:34:50.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 512 MiB in 128 objects 2024-09-14T05:34:50.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:34:50.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T05:34:50.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10a6890e1eb4 2024-09-14T05:34:50.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10a6890e1eb4 2024-09-14T05:34:50.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:34:50.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T05:34:50.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:34:50.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:34:50.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:34:49 2024 2024-09-14T05:34:50.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:34:49 2024 2024-09-14T05:34:50.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:34:49 2024 2024-09-14T05:34:50.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:34:50.604 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:34:50.604 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 77511074-da27-44b2-8b96-37f8b827a31e 2024-09-14T05:34:50.604 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T05:34:50.606 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-14T05:34:50.606 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:34:50.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:34:50.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:34:50.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-14T05:34:50.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:34:50.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:34:50.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:34:50.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:34:50.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:34:50.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:34:50.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:34:51.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-14T05:34:51.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:34:51.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:34:51.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:34:51.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:34:51.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:34:51.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:34:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-14T05:34:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:34:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:34:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:34:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:34:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:34:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:34:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:34:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:34:51.612 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:34:51.612 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:34:51.612 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:34:51.612 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-14T05:34:51.612 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:34:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:34:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:34:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:34:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:34:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:34:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:34:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:34:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:34:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:34:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:34:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:34:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:34:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:34:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:34:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:34:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:34:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:34:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:34:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:34:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:34:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:34:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:34:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:34:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:34:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:34:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:34:51.615 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:34:51.615 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:34:51.615 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:34:51.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:34:51.636 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:34:51.636 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:34:51.636 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-14T05:34:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T05:34:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:34:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:34:51.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:34:51.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-14T05:34:51.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-14T05:34:51.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:34:51.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:34:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T05:34:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-14T05:34:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:34:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:34:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:34:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-14T05:34:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T05:34:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:34:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:34:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-14T05:34:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:34:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T05:34:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-14T05:34:51.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:34:51.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:34:51.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T05:34:51.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-14T05:34:51.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:34:51.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:34:51.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:34:51.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:34:51.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T05:34:51.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:34:51.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:34:51.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-14T05:34:51.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-14T05:34:51.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T05:34:51.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:34:51.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:34:51.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:34:51.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-14T05:34:51.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T05:34:51.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:34:51.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:34:51.808 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:34:51.808 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-14T05:34:51.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-14T05:34:51.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:34:51.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:34:51.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T05:34:51.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-14T05:34:51.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:34:51.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:34:51.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:34:51.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-14T05:34:51.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T05:34:51.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:34:51.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:34:51.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-14T05:34:51.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:34:51.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T05:34:51.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-14T05:34:51.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:34:51.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:34:51.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T05:34:51.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-14T05:34:51.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:34:51.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:34:51.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:34:51.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:34:51.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T05:34:51.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:34:51.811 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:34:51.811 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-14T05:34:51.811 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-14T05:34:51.811 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T05:34:51.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:34:51.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:34:51.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:34:51.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-14T05:34:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T05:34:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:34:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:34:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:34:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-14T05:34:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:3 2024-09-14T05:34:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:34:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:34:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-14T05:34:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:3 2024-09-14T05:34:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:34:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:34:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:34:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=3 2024-09-14T05:34:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-14T05:34:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:34:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:34:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=3 2024-09-14T05:34:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-14T05:34:51.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-14T05:34:51.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:3 2024-09-14T05:34:51.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:34:51.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:34:51.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-14T05:34:51.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:3 2024-09-14T05:34:51.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:34:51.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:34:51.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:34:51.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-14T05:34:51.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-14T05:34:51.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:34:51.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:34:51.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=3 2024-09-14T05:34:51.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=3 2024-09-14T05:34:51.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-14T05:34:51.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T05:34:51.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T05:34:51.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T05:34:51.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-14T05:34:51.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T05:34:51.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:34:51.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:34:51.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:34:51.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 1 2024-09-14T05:34:52.991 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:34:52.991 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:34:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:34:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-14T05:34:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:34:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:34:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:34:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:34:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:34:52.995 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:34:52.995 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:34:52.995 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:34:52.995 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:34:52.995 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:34:52.995 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:34:52.995 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:34:52.995 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:34:52.995 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:34:52.995 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:34:52.995 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:34:52.995 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:34:52.995 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:34:52.995 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:34:52.996 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:34:52.996 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:34:52.996 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:34:52.996 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:34:52.996 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:34:52.996 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:34:52.996 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:34:52.996 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:34:52.996 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:34:52.996 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:34:52.996 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:34:53.018 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:34:53.018 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:34:53.018 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:34:53.018 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-14T05:34:53.095 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T05:34:53.095 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:34:53.095 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:34:53.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:34:53.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-14T05:34:53.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-14T05:34:53.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:34:53.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:34:53.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T05:34:53.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-14T05:34:53.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:34:53.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:34:53.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:34:53.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-14T05:34:53.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T05:34:53.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:34:53.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:34:53.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-14T05:34:53.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:34:53.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T05:34:53.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-14T05:34:53.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:34:53.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:34:53.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T05:34:53.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-14T05:34:53.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:34:53.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:34:53.100 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:34:53.100 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:34:53.100 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T05:34:53.100 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:34:53.100 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:34:53.100 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-14T05:34:53.100 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-14T05:34:53.100 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T05:34:53.117 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:34:53.117 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:34:53.117 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:34:53.117 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-14T05:34:53.186 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T05:34:53.186 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:34:53.187 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:34:53.190 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:34:53.190 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-14T05:34:53.190 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-14T05:34:53.190 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:34:53.190 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:34:53.190 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T05:34:53.190 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-14T05:34:53.190 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:34:53.190 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:34:53.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:34:53.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-14T05:34:53.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T05:34:53.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:34:53.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:34:53.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-14T05:34:53.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:34:53.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T05:34:53.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-14T05:34:53.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:34:53.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:34:53.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T05:34:53.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-14T05:34:53.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:34:53.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:34:53.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:34:53.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:34:53.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T05:34:53.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:34:53.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:34:53.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-14T05:34:53.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-14T05:34:53.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T05:34:53.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:34:53.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:34:53.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:34:53.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-14T05:34:53.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T05:34:53.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:34:53.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:34:53.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:34:53.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-14T05:34:53.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:3 2024-09-14T05:34:53.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:34:53.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:34:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-14T05:34:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:3 2024-09-14T05:34:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:34:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:34:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:34:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=3 2024-09-14T05:34:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-14T05:34:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:34:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:34:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=3 2024-09-14T05:34:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-14T05:34:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-14T05:34:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:3 2024-09-14T05:34:53.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:34:53.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:34:53.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-14T05:34:53.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:3 2024-09-14T05:34:53.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:34:53.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:34:53.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:34:53.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-14T05:34:53.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-14T05:34:53.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:34:53.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:34:53.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=3 2024-09-14T05:34:53.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=3 2024-09-14T05:34:53.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-14T05:34:53.301 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T05:34:53.301 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T05:34:53.301 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T05:34:53.301 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-14T05:34:53.371 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T05:34:53.371 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:34:53.371 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:34:53.371 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:34:53.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 2 2024-09-14T05:34:53.435 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6441267268015436) > weight (0.25) 2024-09-14T05:34:53.435 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.3846075091149127) > weight (0.25) 2024-09-14T05:34:53.436 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.43103884512751234) > weight (0.25) 2024-09-14T05:34:53.436 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.43676073835218443) > weight (0.25) 2024-09-14T05:34:53.436 INFO:tasks.rbd_mirror_thrash:waiting for 85 secs before thrashing 2024-09-14T05:34:55.373 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:34:55.373 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:34:55.376 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:34:55.376 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-14T05:34:55.376 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:34:55.376 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:34:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:34:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:34:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:34:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:34:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:34:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:34:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:34:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:34:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:34:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:34:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:34:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:34:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:34:55.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:34:55.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:34:55.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:34:55.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:34:55.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:34:55.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:34:55.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:34:55.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:34:55.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:34:55.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:34:55.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:34:55.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:34:55.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:34:55.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:34:55.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:34:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:34:55.399 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:34:55.399 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:34:55.399 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-14T05:34:55.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T05:34:55.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:34:55.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:34:55.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:34:55.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-14T05:34:55.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-14T05:34:55.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:34:55.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:34:55.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T05:34:55.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-14T05:34:55.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:34:55.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:34:55.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:34:55.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-14T05:34:55.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T05:34:55.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:34:55.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:34:55.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-14T05:34:55.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:34:55.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T05:34:55.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-14T05:34:55.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:34:55.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:34:55.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T05:34:55.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-14T05:34:55.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:34:55.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:34:55.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:34:55.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:34:55.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T05:34:55.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:34:55.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:34:55.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-14T05:34:55.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-14T05:34:55.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T05:34:55.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:34:55.498 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:34:55.498 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:34:55.498 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-14T05:34:55.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T05:34:55.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:34:55.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:34:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:34:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-14T05:34:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-14T05:34:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:34:55.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:34:55.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T05:34:55.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-14T05:34:55.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:34:55.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:34:55.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:34:55.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-14T05:34:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T05:34:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:34:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:34:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-14T05:34:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:34:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T05:34:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-14T05:34:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:34:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:34:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T05:34:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-14T05:34:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:34:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:34:55.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:34:55.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:34:55.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T05:34:55.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:34:55.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:34:55.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-14T05:34:55.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-14T05:34:55.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T05:34:55.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:34:55.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:34:55.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:34:55.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-14T05:34:55.660 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T05:34:55.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:34:55.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:34:55.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:34:55.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-14T05:34:55.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:3 2024-09-14T05:34:55.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:34:55.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:34:55.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-14T05:34:55.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:3 2024-09-14T05:34:55.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:34:55.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:34:55.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:34:55.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=3 2024-09-14T05:34:55.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-14T05:34:55.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:34:55.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:34:55.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=3 2024-09-14T05:34:55.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-14T05:34:55.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-14T05:34:55.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:3 2024-09-14T05:34:55.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:34:55.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:34:55.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-14T05:34:55.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:3 2024-09-14T05:34:55.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:34:55.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:34:55.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:34:55.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-14T05:34:55.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-14T05:34:55.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:34:55.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:34:55.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=3 2024-09-14T05:34:55.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=3 2024-09-14T05:34:55.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-14T05:34:55.682 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T05:34:55.682 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T05:34:55.682 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T05:34:55.682 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-14T05:34:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T05:34:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:34:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:34:55.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:34:55.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 4 2024-09-14T05:34:59.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:34:59.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:34:59.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:34:59.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-14T05:34:59.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:34:59.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:34:59.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:34:59.761 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:34:59.761 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:34:59.761 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:34:59.761 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:34:59.761 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:34:59.761 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:34:59.762 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:34:59.762 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:34:59.762 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:34:59.762 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:34:59.762 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:34:59.762 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:34:59.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:34:59.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:34:59.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:34:59.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:34:59.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:34:59.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:34:59.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:34:59.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:34:59.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:34:59.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:34:59.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:34:59.765 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:34:59.765 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:34:59.765 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:34:59.765 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:34:59.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:34:59.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:34:59.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:34:59.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-14T05:34:59.856 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T05:34:59.856 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:34:59.856 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:34:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:34:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-14T05:34:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-14T05:34:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:34:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:34:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T05:34:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-14T05:34:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:34:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:34:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:34:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-14T05:34:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T05:34:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:34:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:34:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-14T05:34:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:34:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T05:34:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-14T05:34:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:34:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:34:59.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T05:34:59.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-14T05:34:59.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:34:59.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:34:59.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:34:59.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:34:59.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T05:34:59.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:34:59.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:34:59.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-14T05:34:59.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-14T05:34:59.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T05:34:59.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:34:59.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:34:59.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:34:59.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-14T05:34:59.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T05:34:59.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:34:59.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:34:59.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:34:59.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-14T05:34:59.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-14T05:34:59.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:34:59.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:34:59.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T05:34:59.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-14T05:34:59.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:34:59.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:34:59.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:34:59.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-14T05:34:59.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T05:34:59.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:34:59.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:34:59.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-14T05:34:59.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:34:59.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T05:34:59.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-14T05:34:59.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:34:59.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:34:59.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T05:34:59.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-14T05:34:59.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:34:59.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:34:59.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:34:59.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:34:59.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T05:34:59.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:34:59.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:34:59.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-14T05:34:59.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-14T05:34:59.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T05:34:59.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:34:59.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:34:59.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:34:59.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-14T05:35:00.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T05:35:00.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:35:00.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:35:00.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:35:00.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-14T05:35:00.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:3 2024-09-14T05:35:00.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:35:00.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:35:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-14T05:35:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:3 2024-09-14T05:35:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:35:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:35:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:35:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=3 2024-09-14T05:35:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-14T05:35:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:35:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:35:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=3 2024-09-14T05:35:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-14T05:35:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-14T05:35:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:3 2024-09-14T05:35:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:35:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:35:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-14T05:35:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:3 2024-09-14T05:35:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:35:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:35:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:35:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-14T05:35:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-14T05:35:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:35:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:35:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=3 2024-09-14T05:35:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=3 2024-09-14T05:35:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-14T05:35:00.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T05:35:00.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T05:35:00.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T05:35:00.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-14T05:35:00.128 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T05:35:00.128 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:35:00.128 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:35:00.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:35:00.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 8 2024-09-14T05:35:08.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:35:08.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:35:08.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:35:08.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-14T05:35:08.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:35:08.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:35:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:35:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:35:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:35:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:35:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:35:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:35:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:35:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:35:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:35:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:35:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:35:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:35:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:35:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:35:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:35:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:35:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:35:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:35:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:35:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:35:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:35:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:35:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:35:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:35:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:35:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:35:08.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:35:08.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:35:08.157 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:35:08.157 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:35:08.158 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:35:08.158 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-14T05:35:08.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T05:35:08.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:35:08.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:35:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:35:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-14T05:35:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-14T05:35:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:35:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:35:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T05:35:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-14T05:35:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:35:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:35:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:35:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-14T05:35:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T05:35:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:35:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:35:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-14T05:35:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:35:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T05:35:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-14T05:35:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:35:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:35:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T05:35:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-14T05:35:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:35:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:35:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:35:08.245 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:35:08.245 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T05:35:08.245 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:35:08.245 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:35:08.245 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-14T05:35:08.245 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-14T05:35:08.245 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T05:35:08.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:35:08.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:35:08.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:35:08.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-14T05:35:08.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T05:35:08.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:35:08.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:35:08.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:35:08.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-14T05:35:08.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-14T05:35:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:35:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:35:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T05:35:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-14T05:35:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:35:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:35:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:35:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-14T05:35:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T05:35:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:35:08.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:35:08.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-14T05:35:08.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:35:08.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T05:35:08.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-14T05:35:08.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:35:08.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:35:08.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T05:35:08.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-14T05:35:08.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:35:08.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:35:08.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:35:08.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:35:08.340 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T05:35:08.340 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:35:08.340 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:35:08.340 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-14T05:35:08.340 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-14T05:35:08.340 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T05:35:08.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:35:08.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:35:08.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:35:08.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-14T05:35:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:35:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:35:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:35:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:35:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:35:08.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:35:08.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:35:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:35:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ clean_snap_name= 2024-09-14T05:35:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ seq 1 10 2024-09-14T05:35:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-14T05:35:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror test 2024-09-14T05:35:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:35:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:35:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:35:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:35:08.441 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=8 2024-09-14T05:35:08.441 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:35:08.441 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-14T05:35:08.601 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 2343936 bytes 2024-09-14T05:35:08.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 81081344 bytes 2024-09-14T05:35:08.798 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 110589952 bytes 2024-09-14T05:35:08.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 187087872 bytes 2024-09-14T05:35:08.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 220598272 bytes 2024-09-14T05:35:09.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 271971328 bytes 2024-09-14T05:35:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 351820800 bytes 2024-09-14T05:35:09.220 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 372118528 bytes 2024-09-14T05:35:09.293 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 400957440 bytes 2024-09-14T05:35:09.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 432357376 bytes 2024-09-14T05:35:09.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 470125568 bytes 2024-09-14T05:35:09.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 486223872 bytes 2024-09-14T05:35:09.542 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 490397696 bytes 2024-09-14T05:35:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 540163072 bytes 2024-09-14T05:35:09.719 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 591773696 bytes 2024-09-14T05:35:09.830 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 658639872 bytes 2024-09-14T05:35:09.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 707246080 bytes 2024-09-14T05:35:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 713911296 bytes 2024-09-14T05:35:10.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 750459904 bytes 2024-09-14T05:35:10.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 797088768 bytes 2024-09-14T05:35:10.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 845575168 bytes 2024-09-14T05:35:10.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 898885632 bytes 2024-09-14T05:35:10.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 953915392 bytes 2024-09-14T05:35:10.542 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1018013696 bytes 2024-09-14T05:35:10.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1022746624 bytes 2024-09-14T05:35:10.686 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1072815104 bytes 2024-09-14T05:35:10.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1136794624 bytes 2024-09-14T05:35:10.900 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1212223488 bytes 2024-09-14T05:35:10.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1235770368 bytes 2024-09-14T05:35:11.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1270821888 bytes 2024-09-14T05:35:11.132 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1283615744 bytes 2024-09-14T05:35:11.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1300286464 bytes 2024-09-14T05:35:11.270 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1300918272 bytes 2024-09-14T05:35:11.340 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1314570240 bytes 2024-09-14T05:35:11.401 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1315179520 bytes 2024-09-14T05:35:11.538 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1399050240 bytes 2024-09-14T05:35:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1418656768 bytes 2024-09-14T05:35:11.704 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1444984832 bytes 2024-09-14T05:35:11.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1461901312 bytes 2024-09-14T05:35:11.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1502509056 bytes 2024-09-14T05:35:11.932 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1503127552 bytes 2024-09-14T05:35:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1570426880 bytes 2024-09-14T05:35:12.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1597977600 bytes 2024-09-14T05:35:12.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1621284864 bytes 2024-09-14T05:35:12.322 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1692881920 bytes 2024-09-14T05:35:12.403 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1706493952 bytes 2024-09-14T05:35:12.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 1734625280 bytes 2024-09-14T05:35:12.619 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 1816508416 bytes 2024-09-14T05:35:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 1916943360 bytes 2024-09-14T05:35:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2010348544 bytes 2024-09-14T05:35:12.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2024792064 bytes 2024-09-14T05:35:13.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2092607488 bytes 2024-09-14T05:35:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2109707264 bytes 2024-09-14T05:35:13.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2201325568 bytes 2024-09-14T05:35:13.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2309818368 bytes 2024-09-14T05:35:13.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2315860992 bytes 2024-09-14T05:35:13.623 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2377105408 bytes 2024-09-14T05:35:13.696 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2385210368 bytes 2024-09-14T05:35:13.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2385819648 bytes 2024-09-14T05:35:13.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2436270080 bytes 2024-09-14T05:35:13.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2441784320 bytes 2024-09-14T05:35:13.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2445278208 bytes 2024-09-14T05:35:14.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2537667584 bytes 2024-09-14T05:35:14.229 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2597675008 bytes 2024-09-14T05:35:14.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2612108288 bytes 2024-09-14T05:35:14.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 2640154624 bytes 2024-09-14T05:35:14.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 2655715328 bytes 2024-09-14T05:35:14.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 2678752256 bytes 2024-09-14T05:35:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 2679373824 bytes 2024-09-14T05:35:14.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 2766724096 bytes 2024-09-14T05:35:14.858 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 2858924032 bytes 2024-09-14T05:35:14.911 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 2863159296 bytes 2024-09-14T05:35:15.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7300 IOs, 2900234240 bytes 2024-09-14T05:35:15.101 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7400 IOs, 2945388544 bytes 2024-09-14T05:35:15.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7500 IOs, 2945991680 bytes 2024-09-14T05:35:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7600 IOs, 2993783808 bytes 2024-09-14T05:35:15.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7700 IOs, 3019898880 bytes 2024-09-14T05:35:15.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7800 IOs, 3020520448 bytes 2024-09-14T05:35:15.550 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7900 IOs, 3113402368 bytes 2024-09-14T05:35:15.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8000 IOs, 3163919360 bytes 2024-09-14T05:35:15.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8100 IOs, 3193637888 bytes 2024-09-14T05:35:15.811 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8200 IOs, 3218704384 bytes 2024-09-14T05:35:15.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8300 IOs, 3275787264 bytes 2024-09-14T05:35:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8400 IOs, 3302353920 bytes 2024-09-14T05:35:16.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8500 IOs, 3328343040 bytes 2024-09-14T05:35:16.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8600 IOs, 3347512320 bytes 2024-09-14T05:35:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8700 IOs, 3348107264 bytes 2024-09-14T05:35:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8800 IOs, 3368740864 bytes 2024-09-14T05:35:16.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8900 IOs, 3370592256 bytes 2024-09-14T05:35:16.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9000 IOs, 3383598080 bytes 2024-09-14T05:35:16.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:35:16.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:35:16.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:35:16.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:35:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-14T05:35:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:35:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:35:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:35:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-14T05:35:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local description_pattern= 2024-09-14T05:35:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local service_pattern= 2024-09-14T05:35:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:35:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:35:17.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-14T05:35:17.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:35:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:35:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:35:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-14T05:35:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local description_pattern= 2024-09-14T05:35:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local service_pattern= 2024-09-14T05:35:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status 2024-09-14T05:35:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CEPH_ARGS= 2024-09-14T05:35:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-14T05:35:17.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status='test: 2024-09-14T05:35:17.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: global_id: 77511074-da27-44b2-8b96-37f8b827a31e 2024-09-14T05:35:17.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: state: down+unknown 2024-09-14T05:35:17.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: description: status not found 2024-09-14T05:35:17.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: last_update: 2024-09-14T05:35:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: peer_sites: 2024-09-14T05:35:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: name: cluster2 2024-09-14T05:35:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: state: down+unknown 2024-09-14T05:35:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: description: status not found 2024-09-14T05:35:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: last_update: ' 2024-09-14T05:35:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^ state: .*up+replaying' 2024-09-14T05:35:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 1 2024-09-14T05:35:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:35:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 2 2024-09-14T05:35:19.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-14T05:35:19.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:35:19.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:35:19.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:35:19.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-14T05:35:19.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local description_pattern= 2024-09-14T05:35:19.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local service_pattern= 2024-09-14T05:35:19.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status 2024-09-14T05:35:19.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CEPH_ARGS= 2024-09-14T05:35:19.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-14T05:35:19.563 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status='test: 2024-09-14T05:35:19.563 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: global_id: 77511074-da27-44b2-8b96-37f8b827a31e 2024-09-14T05:35:19.563 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: state: down+unknown 2024-09-14T05:35:19.563 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: description: status not found 2024-09-14T05:35:19.563 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: last_update: 2024-09-14T05:35:19.563 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: peer_sites: 2024-09-14T05:35:19.563 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: name: cluster2 2024-09-14T05:35:19.563 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: state: down+unknown 2024-09-14T05:35:19.563 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: description: status not found 2024-09-14T05:35:19.563 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: last_update: ' 2024-09-14T05:35:19.563 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^ state: .*up+replaying' 2024-09-14T05:35:19.564 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 1 2024-09-14T05:35:19.564 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:35:19.564 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 4 2024-09-14T05:35:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-14T05:35:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:35:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:35:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:35:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-14T05:35:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local description_pattern= 2024-09-14T05:35:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local service_pattern= 2024-09-14T05:35:23.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status 2024-09-14T05:35:23.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CEPH_ARGS= 2024-09-14T05:35:23.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-14T05:35:23.621 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status='test: 2024-09-14T05:35:23.621 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: global_id: 77511074-da27-44b2-8b96-37f8b827a31e 2024-09-14T05:35:23.621 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: state: down+unknown 2024-09-14T05:35:23.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: description: status not found 2024-09-14T05:35:23.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: last_update: 2024-09-14T05:35:23.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: peer_sites: 2024-09-14T05:35:23.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: name: cluster2 2024-09-14T05:35:23.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: state: down+unknown 2024-09-14T05:35:23.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: description: status not found 2024-09-14T05:35:23.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: last_update: ' 2024-09-14T05:35:23.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^ state: .*up+replaying' 2024-09-14T05:35:23.623 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 1 2024-09-14T05:35:23.623 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:35:23.623 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 8 2024-09-14T05:35:26.435 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.2668426298121688) > weight (0.25) 2024-09-14T05:35:26.435 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.4279419802167508) > weight (0.25) 2024-09-14T05:35:26.435 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.9099010370002204) > weight (0.25) 2024-09-14T05:35:26.435 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.7113123020543877) > weight (0.25) 2024-09-14T05:35:26.436 INFO:tasks.rbd_mirror_thrash:waiting for 68 secs before thrashing 2024-09-14T05:35:31.624 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-14T05:35:31.624 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:35:31.625 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:35:31.625 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:35:31.625 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-14T05:35:31.625 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local description_pattern= 2024-09-14T05:35:31.625 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local service_pattern= 2024-09-14T05:35:31.625 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status 2024-09-14T05:35:31.625 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CEPH_ARGS= 2024-09-14T05:35:31.625 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-14T05:35:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status='test: 2024-09-14T05:35:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: global_id: 77511074-da27-44b2-8b96-37f8b827a31e 2024-09-14T05:35:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: state: down+unknown 2024-09-14T05:35:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: description: status not found 2024-09-14T05:35:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: last_update: 2024-09-14T05:35:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: peer_sites: 2024-09-14T05:35:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: name: cluster2 2024-09-14T05:35:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: state: down+unknown 2024-09-14T05:35:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: description: status not found 2024-09-14T05:35:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: last_update: ' 2024-09-14T05:35:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^ state: .*up+replaying' 2024-09-14T05:35:31.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 1 2024-09-14T05:35:31.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:35:31.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 8 2024-09-14T05:35:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-14T05:35:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:35:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:35:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:35:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-14T05:35:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local description_pattern= 2024-09-14T05:35:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local service_pattern= 2024-09-14T05:35:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status 2024-09-14T05:35:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CEPH_ARGS= 2024-09-14T05:35:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-14T05:35:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status='test: 2024-09-14T05:35:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: global_id: 77511074-da27-44b2-8b96-37f8b827a31e 2024-09-14T05:35:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: state: up+replaying 2024-09-14T05:35:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726292090,"remote_snapshot_timestamp":1726292090,"replay_state":"idle"} 2024-09-14T05:35:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: service: mirror.2 on smithi191 2024-09-14T05:35:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: last_update: 2024-09-14 05:35:35 2024-09-14T05:35:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: peer_sites: 2024-09-14T05:35:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: name: cluster2 2024-09-14T05:35:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: state: up+stopped 2024-09-14T05:35:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: description: local image is primary 2024-09-14T05:35:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: last_update: 2024-09-14 05:35:35' 2024-09-14T05:35:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^ state: .*up+replaying' 2024-09-14T05:35:39.740 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: state: up+replaying 2024-09-14T05:35:39.740 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^ description: .*' 2024-09-14T05:35:39.740 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726292090,"remote_snapshot_timestamp":1726292090,"replay_state":"idle"} 2024-09-14T05:35:39.741 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:35:39.741 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo up+replaying 2024-09-14T05:35:39.741 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^up+' 2024-09-14T05:35:39.742 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:up+replaying 2024-09-14T05:35:39.742 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep 'service: *mirror..* on ' 2024-09-14T05:35:39.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: service: mirror.2 on smithi191 2024-09-14T05:35:39.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local last_update 2024-09-14T05:35:39.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-14T05:35:39.745 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ last_update='2024-09-14 05:35:35' 2024-09-14T05:35:39.745 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-14 05:35:35' 2024-09-14T05:35:39.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:35:39.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:35:39.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:35:39.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-14T05:35:39.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local 'prev_last_update=2024-09-14 05:35:35' 2024-09-14T05:35:39.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status 2024-09-14T05:35:39.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CEPH_ARGS= 2024-09-14T05:35:39.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-14T05:35:39.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status='OKOKOK142124293mirror.0smithi19119.3.0-4775-g7f24e589falseOK42164296mirror.1smithi19119.3.0-4775-g7f24e589falseOK42204289mirror.2smithi19119.3.0-4775-g7f24e589trueOK42244292mirror.3smithi19119.3.0-4775-g7f24e589falseOKtest77511074-da27-44b2-8b96-37f8b827a31eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726292090,"remote_snapshot_timestamp":1726292090,"replay_state":"idle"}42204289mirror.2smithi1912024-09-14 05:35:35cluster29d38de88-cffd-4132-b7e8-809c69f2745aup+stoppedlocal image is primary2024-09-14 05:35:35' 2024-09-14T05:35:39.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local last_update state 2024-09-14T05:35:39.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-14T05:35:39.844 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ last_update='2024-09-14 05:35:35' 2024-09-14T05:35:39.844 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-14T05:35:39.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ state=up+replaying 2024-09-14T05:35:39.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo up+replaying 2024-09-14T05:35:39.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep up+replaying 2024-09-14T05:35:39.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:up+replaying 2024-09-14T05:35:39.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:35:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:35:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap1 2024-09-14T05:35:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror test snap1 2024-09-14T05:35:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:35:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:35:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:35:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap1 2024-09-14T05:35:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:35:46.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-14T05:35:46.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:35:46.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:35:46.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:35:46.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-14T05:35:46.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:35:46.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:35:46.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:35:46.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:35:46.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:35:46.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:35:46.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:35:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-14T05:35:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:35:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:35:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:35:47.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:35:47.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:35:47.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:35:47.489 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-14T05:35:47.489 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:35:47.489 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:35:47.489 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:35:47.489 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:35:47.489 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:35:47.489 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:35:47.489 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:35:47.489 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:35:47.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:35:47.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:35:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:35:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-14T05:35:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:35:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:35:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:35:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:35:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:35:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:35:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:35:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:35:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:35:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:35:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:35:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:35:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:35:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:35:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:35:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:35:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:35:47.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:35:47.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:35:47.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:35:47.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:35:47.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:35:47.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:35:47.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:35:47.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:35:47.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:35:47.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:35:47.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:35:47.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:35:47.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:35:47.514 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:35:47.514 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:35:47.514 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:35:47.514 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-14T05:35:47.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T05:35:47.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:35:47.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:35:47.596 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:35:47.596 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-14T05:35:47.596 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-14T05:35:47.596 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:35:47.596 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:35:47.596 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T05:35:47.596 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-14T05:35:47.596 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:35:47.596 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:35:47.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:35:47.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-14T05:35:47.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T05:35:47.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:35:47.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:35:47.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-14T05:35:47.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:35:47.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T05:35:47.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-14T05:35:47.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:35:47.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:35:47.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T05:35:47.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-14T05:35:47.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:35:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:35:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:35:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:35:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T05:35:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:35:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:35:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-14T05:35:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-14T05:35:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T05:35:47.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:35:47.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:35:47.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:35:47.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-14T05:35:47.689 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T05:35:47.689 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:35:47.689 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:35:47.692 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:35:47.692 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-14T05:35:47.692 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-14T05:35:47.692 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:35:47.692 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:35:47.692 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T05:35:47.692 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-14T05:35:47.692 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:35:47.692 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:35:47.693 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:35:47.693 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-14T05:35:47.693 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T05:35:47.693 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:35:47.693 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:35:47.693 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-14T05:35:47.693 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:35:47.693 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T05:35:47.693 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-14T05:35:47.693 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:35:47.693 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:35:47.693 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T05:35:47.693 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-14T05:35:47.694 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:35:47.694 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:35:47.694 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:35:47.694 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:35:47.694 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T05:35:47.694 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:35:47.694 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:35:47.694 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-14T05:35:47.694 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-14T05:35:47.694 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T05:35:47.710 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:35:47.710 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:35:47.710 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:35:47.711 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-14T05:35:47.791 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:35:47.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:35:47.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:35:47.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:35:47.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:35:47.794 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:35:47.794 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:35:47.794 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:35:47.795 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-14T05:35:47.795 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:35:47.795 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:35:47.795 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:35:47.795 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:35:47.795 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:35:47.795 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:35:47.795 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-14T05:35:47.795 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:35:47.795 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:35:47.795 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:35:47.795 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:35:47.795 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-test.status 2024-09-14T05:35:47.795 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-test.status 2024-09-14T05:35:47.796 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:35:47.797 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.status 2024-09-14T05:35:47.797 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-test.status 2024-09-14T05:35:47.797 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-test.status 2024-09-14T05:35:47.797 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:35:47.797 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:35:47.797 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-14T05:35:47.797 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:35:47.797 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:35:47.797 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:35:47.797 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-14T05:35:48.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 5 2024-09-14T05:35:48.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.status 2024-09-14T05:35:48.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:35:48.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:35:48.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:35:48.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.status 2024-09-14T05:35:48.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-14T05:35:48.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:35:48.542 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:35:48.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=5 2024-09-14T05:35:48.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:35:48.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:35:48.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:35:48.745 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:35:48.745 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:35:48.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:35:48.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:35:48.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:35:48.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T05:35:48.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:35:48.796 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:35:48.797 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=3 2024-09-14T05:35:48.798 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 5 = 3 2024-09-14T05:35:48.798 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:35:48.798 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:35:49.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:35:49.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:35:49.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:35:49.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:35:49.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:35:49.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T05:35:49.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:35:49.251 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:35:49.251 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:35:49.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:35:49.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:35:49.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:35:49.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:35:49.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 5 = '' 2024-09-14T05:35:49.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:35:49.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:35:50.055 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:35:50.055 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:35:50.056 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:35:50.056 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:35:50.056 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:35:50.056 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T05:35:50.056 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:35:50.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:35:50.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:35:50.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:35:50.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:35:50.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:35:50.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:35:50.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 5 = '' 2024-09-14T05:35:50.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:35:50.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-14T05:35:51.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:35:51.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:35:51.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:35:51.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:35:51.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:35:51.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T05:35:51.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:35:51.781 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:35:51.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=5 2024-09-14T05:35:51.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 5 = 5 2024-09-14T05:35:51.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:35:51.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror test snap1 2024-09-14T05:35:51.784 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:35:51.784 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:35:51.784 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:35:51.784 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap1 2024-09-14T05:35:51.784 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:35:51.784 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:35:51.784 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:35:52.784 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/test@snap1 2024-09-14T05:35:52.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'test': 2024-09-14T05:35:52.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 512 MiB in 128 objects 2024-09-14T05:35:52.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:35:52.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-14T05:35:52.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c1a41659ac 2024-09-14T05:35:52.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c1a41659ac 2024-09-14T05:35:52.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:35:52.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:35:52.835 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:35:52.835 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:35:52.835 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:35:08 2024 2024-09-14T05:35:52.835 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:35:08 2024 2024-09-14T05:35:52.835 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:35:08 2024 2024-09-14T05:35:52.835 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:35:52.835 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:35:52.835 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:35:52.835 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 77511074-da27-44b2-8b96-37f8b827a31e 2024-09-14T05:35:52.835 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:35:52.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:35:52.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:35:52.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap1 2024-09-14T05:35:52.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:35:52.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:35:52.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap1 2024-09-14T05:35:52.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:35:52.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:35:52.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:35:52.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:35:52.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:35:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:35:53.375 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:35:54.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:35:54.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:35:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:35:54.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:35:54.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ clean_snap_name=snap1-clean 2024-09-14T05:35:54.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror test snap1-clean 2024-09-14T05:35:54.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:35:54.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:35:54.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:35:54.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap1-clean 2024-09-14T05:35:54.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1-clean --debug-rbd=20 --debug-journaler=20 2024-09-14T05:35:54.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-14T05:35:54.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror test 2024-09-14T05:35:54.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:35:54.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:35:54.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:35:54.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:35:54.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=9 2024-09-14T05:35:54.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:35:54.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-14T05:35:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 616448 bytes 2024-09-14T05:35:54.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 65160192 bytes 2024-09-14T05:35:54.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 108365824 bytes 2024-09-14T05:35:55.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 168226816 bytes 2024-09-14T05:35:55.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 174749696 bytes 2024-09-14T05:35:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 237819904 bytes 2024-09-14T05:35:55.396 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 282983424 bytes 2024-09-14T05:35:55.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 338188288 bytes 2024-09-14T05:35:55.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 451592192 bytes 2024-09-14T05:35:55.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 491548672 bytes 2024-09-14T05:35:55.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 502018048 bytes 2024-09-14T05:35:55.931 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 520475648 bytes 2024-09-14T05:35:56.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 599437312 bytes 2024-09-14T05:35:56.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 610398208 bytes 2024-09-14T05:35:56.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 611030016 bytes 2024-09-14T05:35:56.264 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 620123136 bytes 2024-09-14T05:35:56.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 673944576 bytes 2024-09-14T05:35:56.436 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 689999872 bytes 2024-09-14T05:35:56.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 725072896 bytes 2024-09-14T05:35:56.619 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 775366656 bytes 2024-09-14T05:35:56.705 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 800754688 bytes 2024-09-14T05:35:56.816 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 876212224 bytes 2024-09-14T05:35:56.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 902612992 bytes 2024-09-14T05:35:56.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 912777216 bytes 2024-09-14T05:35:57.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1026034688 bytes 2024-09-14T05:35:57.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1065344000 bytes 2024-09-14T05:35:57.312 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1085117440 bytes 2024-09-14T05:35:57.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1104390144 bytes 2024-09-14T05:35:57.469 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1142671360 bytes 2024-09-14T05:35:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1143412736 bytes 2024-09-14T05:35:57.646 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1197893632 bytes 2024-09-14T05:35:57.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1216411648 bytes 2024-09-14T05:35:57.815 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1266126848 bytes 2024-09-14T05:35:57.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1313033216 bytes 2024-09-14T05:35:57.978 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1320694784 bytes 2024-09-14T05:35:58.054 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1342021632 bytes 2024-09-14T05:35:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1367989248 bytes 2024-09-14T05:35:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1368602624 bytes 2024-09-14T05:35:58.273 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1403017216 bytes 2024-09-14T05:35:58.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1425354752 bytes 2024-09-14T05:35:58.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1502288896 bytes 2024-09-14T05:35:58.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1573096448 bytes 2024-09-14T05:35:58.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1594725376 bytes 2024-09-14T05:35:58.742 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1610787840 bytes 2024-09-14T05:35:58.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1627532288 bytes 2024-09-14T05:35:58.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1628148736 bytes 2024-09-14T05:35:58.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 1657071616 bytes 2024-09-14T05:35:59.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 1708323840 bytes 2024-09-14T05:35:59.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 1769030656 bytes 2024-09-14T05:35:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 1799172096 bytes 2024-09-14T05:35:59.373 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 1858270208 bytes 2024-09-14T05:35:59.456 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 1891519488 bytes 2024-09-14T05:35:59.552 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 1938167808 bytes 2024-09-14T05:35:59.669 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2010537984 bytes 2024-09-14T05:35:59.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2040847360 bytes 2024-09-14T05:35:59.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2093865984 bytes 2024-09-14T05:35:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2148815872 bytes 2024-09-14T05:36:00.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2173894656 bytes 2024-09-14T05:36:00.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2223592448 bytes 2024-09-14T05:36:00.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2251828224 bytes 2024-09-14T05:36:00.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2306282496 bytes 2024-09-14T05:36:00.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2331975680 bytes 2024-09-14T05:36:00.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2372102144 bytes 2024-09-14T05:36:00.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2431826944 bytes 2024-09-14T05:36:00.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2494846976 bytes 2024-09-14T05:36:00.831 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 2519276544 bytes 2024-09-14T05:36:00.914 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 2542670848 bytes 2024-09-14T05:36:00.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 2549360640 bytes 2024-09-14T05:36:01.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 2553693184 bytes 2024-09-14T05:36:01.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 2601381888 bytes 2024-09-14T05:36:01.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 2629638144 bytes 2024-09-14T05:36:01.352 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 2685545472 bytes 2024-09-14T05:36:01.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7300 IOs, 2720009216 bytes 2024-09-14T05:36:01.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7400 IOs, 2742036480 bytes 2024-09-14T05:36:01.606 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7500 IOs, 2756009984 bytes 2024-09-14T05:36:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7600 IOs, 2757558272 bytes 2024-09-14T05:36:01.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7700 IOs, 2807057408 bytes 2024-09-14T05:36:01.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7800 IOs, 2817294336 bytes 2024-09-14T05:36:01.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7900 IOs, 2906194944 bytes 2024-09-14T05:36:02.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8000 IOs, 2952787968 bytes 2024-09-14T05:36:02.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8100 IOs, 3018404864 bytes 2024-09-14T05:36:02.293 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8200 IOs, 3059258368 bytes 2024-09-14T05:36:02.370 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8300 IOs, 3072724992 bytes 2024-09-14T05:36:02.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8400 IOs, 3077743616 bytes 2024-09-14T05:36:02.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8500 IOs, 3107035136 bytes 2024-09-14T05:36:02.577 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8600 IOs, 3116093440 bytes 2024-09-14T05:36:02.685 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8700 IOs, 3186400256 bytes 2024-09-14T05:36:02.801 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8800 IOs, 3235333120 bytes 2024-09-14T05:36:02.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8900 IOs, 3284826112 bytes 2024-09-14T05:36:03.022 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9000 IOs, 3331100672 bytes 2024-09-14T05:36:03.084 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9100 IOs, 3337352192 bytes 2024-09-14T05:36:03.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9200 IOs, 3367000064 bytes 2024-09-14T05:36:03.322 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9300 IOs, 3471835136 bytes 2024-09-14T05:36:03.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9400 IOs, 3557079040 bytes 2024-09-14T05:36:03.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:36:03.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:36:03.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:36:03.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:36:03.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-14T05:36:03.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:36:03.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:36:03.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:36:03.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-14T05:36:03.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local description_pattern= 2024-09-14T05:36:03.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local service_pattern= 2024-09-14T05:36:03.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:36:03.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:36:04.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-14T05:36:04.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:36:04.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:36:04.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:36:04.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-14T05:36:04.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local description_pattern= 2024-09-14T05:36:04.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local service_pattern= 2024-09-14T05:36:04.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status 2024-09-14T05:36:04.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CEPH_ARGS= 2024-09-14T05:36:04.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-14T05:36:04.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status='test: 2024-09-14T05:36:04.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: global_id: 77511074-da27-44b2-8b96-37f8b827a31e 2024-09-14T05:36:04.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: state: up+replaying 2024-09-14T05:36:04.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726292090,"remote_snapshot_timestamp":1726292090,"replay_state":"idle"} 2024-09-14T05:36:04.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: service: mirror.2 on smithi191 2024-09-14T05:36:04.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: last_update: 2024-09-14 05:35:35 2024-09-14T05:36:04.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: peer_sites: 2024-09-14T05:36:04.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: name: cluster2 2024-09-14T05:36:04.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: state: up+stopped 2024-09-14T05:36:04.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: description: local image is primary 2024-09-14T05:36:04.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: last_update: 2024-09-14 05:35:35' 2024-09-14T05:36:04.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^ state: .*up+replaying' 2024-09-14T05:36:04.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: state: up+replaying 2024-09-14T05:36:04.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^ description: .*' 2024-09-14T05:36:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726292090,"remote_snapshot_timestamp":1726292090,"replay_state":"idle"} 2024-09-14T05:36:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:36:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo up+replaying 2024-09-14T05:36:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^up+' 2024-09-14T05:36:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:up+replaying 2024-09-14T05:36:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep 'service: *mirror..* on ' 2024-09-14T05:36:04.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: service: mirror.2 on smithi191 2024-09-14T05:36:04.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local last_update 2024-09-14T05:36:04.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-14T05:36:04.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ last_update='2024-09-14 05:35:35' 2024-09-14T05:36:04.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-14 05:35:35' 2024-09-14T05:36:04.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:36:04.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:36:04.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:36:04.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-14T05:36:04.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local 'prev_last_update=2024-09-14 05:35:35' 2024-09-14T05:36:04.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status 2024-09-14T05:36:04.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CEPH_ARGS= 2024-09-14T05:36:04.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-14T05:36:04.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status='OKOKOK142124293mirror.0smithi19119.3.0-4775-g7f24e589falseOK42164296mirror.1smithi19119.3.0-4775-g7f24e589falseOK42204289mirror.2smithi19119.3.0-4775-g7f24e589trueOK42244292mirror.3smithi19119.3.0-4775-g7f24e589falseOKtest77511074-da27-44b2-8b96-37f8b827a31eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726292090,"remote_snapshot_timestamp":1726292090,"replay_state":"idle"}42204289mirror.2smithi1912024-09-14 05:35:35cluster29d38de88-cffd-4132-b7e8-809c69f2745aup+stoppedlocal image is primary2024-09-14 05:35:35' 2024-09-14T05:36:04.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local last_update state 2024-09-14T05:36:04.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-14T05:36:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ last_update='2024-09-14 05:35:35' 2024-09-14T05:36:04.637 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-14T05:36:04.638 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ state=up+replaying 2024-09-14T05:36:04.639 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo up+replaying 2024-09-14T05:36:04.639 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep up+replaying 2024-09-14T05:36:04.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:up+replaying 2024-09-14T05:36:04.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:36:04.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:36:04.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap2 2024-09-14T05:36:04.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror test snap2 2024-09-14T05:36:04.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:36:04.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:36:04.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:36:04.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap2 2024-09-14T05:36:04.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:36:18.436 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8264873762031623) > weight (0.25) 2024-09-14T05:36:18.437 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5485331979085392) > weight (0.25) 2024-09-14T05:36:18.437 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.2801655948060914) > weight (0.25) 2024-09-14T05:36:18.437 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.9462243924125352) > weight (0.25) 2024-09-14T05:36:18.437 INFO:tasks.rbd_mirror_thrash:waiting for 93 secs before thrashing 2024-09-14T05:36:33.557 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-14T05:36:33.557 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:36:33.557 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:36:33.558 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:36:33.558 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-14T05:36:33.558 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:36:33.558 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:36:33.558 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:36:33.558 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:36:33.558 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:36:33.558 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:36:33.558 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:36:34.436 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-14T05:36:34.437 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-14T05:36:34.438 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-09-14T05:36:34.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-14T05:36:34.438+0000 7f7b86c2f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 36682) UID: 1000 2024-09-14T05:36:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-14T05:36:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:36:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:36:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:36:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:36:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:36:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:36:34.559 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-14T05:36:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:36:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:36:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:36:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:36:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:36:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:36:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:36:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:36:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:36:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:36:34.562 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:36:34.562 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-14T05:36:34.562 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:36:34.562 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:36:34.562 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:36:34.562 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:36:34.562 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:36:34.562 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:36:34.562 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:36:34.562 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:36:34.563 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:36:34.563 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:36:34.563 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:36:34.563 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:36:34.563 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:36:34.563 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:36:34.563 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:36:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:36:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:36:34.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:36:34.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:36:34.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:36:34.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:36:34.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:36:34.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:36:34.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:36:34.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:36:34.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:36:34.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:36:34.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:36:34.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:36:34.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:36:34.584 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:36:34.584 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:36:34.584 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:36:34.584 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-14T05:36:34.652 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-09-14T05:36:34.655 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:36:34.656 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-14T05:36:34.656 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-14T05:36:34.656 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:36:34.656 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:36:34.656 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T05:36:34.656 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-14T05:36:34.656 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:36:34.656 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:36:34.656 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:36:34.656 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-14T05:36:34.656 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T05:36:34.657 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:36:34.657 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:36:34.657 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-14T05:36:34.657 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:36:34.657 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T05:36:34.657 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-14T05:36:34.657 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:36:34.657 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:36:34.657 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T05:36:34.657 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-14T05:36:34.657 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:36:34.657 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:36:34.657 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:36:34.658 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:36:34.658 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T05:36:34.658 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:36:34.658 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:36:34.658 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-14T05:36:34.658 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-14T05:36:34.658 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T05:36:34.674 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:36:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:36:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:36:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-14T05:36:34.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T05:36:34.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:36:34.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:36:34.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:36:34.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-14T05:36:34.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-14T05:36:34.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:36:34.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:36:34.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T05:36:34.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-14T05:36:34.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:36:34.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:36:34.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:36:34.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-14T05:36:34.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T05:36:34.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:36:34.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:36:34.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-14T05:36:34.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:36:34.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T05:36:34.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-14T05:36:34.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:36:34.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:36:34.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T05:36:34.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-14T05:36:34.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:36:34.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:36:34.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:36:34.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:36:34.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T05:36:34.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:36:34.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:36:34.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-14T05:36:34.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-14T05:36:34.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T05:36:34.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:36:34.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:36:34.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:36:34.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-14T05:36:34.848 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:36:34.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:36:34.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:36:34.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:36:34.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:36:34.851 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:36:34.852 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:36:34.852 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:36:34.852 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-14T05:36:34.852 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:36:34.852 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:36:34.852 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:36:34.852 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:36:34.852 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:36:34.852 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:36:34.852 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-14T05:36:34.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:36:34.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:36:34.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:36:34.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:36:34.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-test.status 2024-09-14T05:36:34.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-test.status 2024-09-14T05:36:34.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:36:34.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.status 2024-09-14T05:36:34.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-test.status 2024-09-14T05:36:34.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-test.status 2024-09-14T05:36:34.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:36:34.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:36:34.855 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-14T05:36:34.855 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:36:34.855 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:36:34.855 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:36:34.855 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-14T05:36:36.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 8 2024-09-14T05:36:36.571 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.status 2024-09-14T05:36:36.571 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:36:36.571 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:36:36.571 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:36:36.571 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.status 2024-09-14T05:36:36.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-14T05:36:36.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:36:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:36:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=8 2024-09-14T05:36:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:36:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:36:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:36:36.821 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:36:36.821 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:36:36.821 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:36:36.822 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:36:36.822 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:36:36.822 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T05:36:36.822 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:36:36.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:36:36.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:36:36.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:36:36.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:36:36.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:36:36.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:36:36.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 8 = '' 2024-09-14T05:36:36.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:36:36.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:36:37.272 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:36:37.272 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:36:37.272 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:36:37.272 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:36:37.272 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:36:37.273 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T05:36:37.273 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:36:37.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:36:37.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:36:37.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:36:37.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:36:37.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:36:37.322 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:36:37.322 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 8 = '' 2024-09-14T05:36:37.322 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:36:37.322 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:36:38.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:36:38.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:36:38.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:36:38.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:36:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:36:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T05:36:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:36:38.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:36:38.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:36:38.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:36:38.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:36:38.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:36:38.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:36:38.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 8 = '' 2024-09-14T05:36:38.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:36:38.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-14T05:36:39.808 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:36:39.808 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:36:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:36:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:36:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:36:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T05:36:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:36:39.863 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:36:39.865 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=8 2024-09-14T05:36:39.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 8 = 8 2024-09-14T05:36:39.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:36:39.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror test snap2 2024-09-14T05:36:39.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:36:39.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:36:39.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:36:39.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap2 2024-09-14T05:36:39.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:36:39.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:36:39.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:36:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/test@snap2 2024-09-14T05:36:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'test': 2024-09-14T05:36:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 512 MiB in 128 objects 2024-09-14T05:36:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:36:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 4 2024-09-14T05:36:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c1a41659ac 2024-09-14T05:36:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c1a41659ac 2024-09-14T05:36:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:36:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:36:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:36:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:36:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:35:08 2024 2024-09-14T05:36:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:35:08 2024 2024-09-14T05:36:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:35:08 2024 2024-09-14T05:36:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:36:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:36:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:36:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 77511074-da27-44b2-8b96-37f8b827a31e 2024-09-14T05:36:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:36:40.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:36:40.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snap1-clean ']' 2024-09-14T05:36:40.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap1-clean 2024-09-14T05:36:40.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:36:40.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:36:40.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap1-clean 2024-09-14T05:36:40.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:36:40.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:36:40.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:36:40.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:36:40.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:36:41.439 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-14T05:36:41.439 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T05:36:41.439 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-14T05:36:41.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-14T05:36:41.440 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-14T05:36:41.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-14T05:36:41.443 INFO:tasks.rbd_mirror_thrash:waiting for 109 secs before thrashing 2024-09-14T05:36:41.460 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:36:41.471 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:36:42.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:36:42.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:36:42.343 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:36:42.459 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:36:42.459 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap2 2024-09-14T05:36:42.459 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:36:42.459 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:36:42.459 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap2 2024-09-14T05:36:42.459 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:36:42.459 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:36:42.460 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:36:42.460 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:36:42.460 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:36:42.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:36:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:36:43.657 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:36:43.667 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:36:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:36:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:36:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ clean_snap_name=snap2-clean 2024-09-14T05:36:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror test snap2-clean 2024-09-14T05:36:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:36:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:36:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:36:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap2-clean 2024-09-14T05:36:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2-clean --debug-rbd=20 --debug-journaler=20 2024-09-14T05:36:44.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-14T05:36:44.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror test 2024-09-14T05:36:44.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:36:44.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:36:44.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:36:44.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:36:44.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=5 2024-09-14T05:36:44.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:36:44.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-14T05:36:44.787 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 17619968 bytes 2024-09-14T05:36:44.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 38713344 bytes 2024-09-14T05:36:45.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 96745472 bytes 2024-09-14T05:36:45.128 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 119955456 bytes 2024-09-14T05:36:45.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 148942848 bytes 2024-09-14T05:36:45.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 170411008 bytes 2024-09-14T05:36:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 188262400 bytes 2024-09-14T05:36:45.499 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 254647296 bytes 2024-09-14T05:36:45.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 300971008 bytes 2024-09-14T05:36:45.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 307471360 bytes 2024-09-14T05:36:45.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 363737088 bytes 2024-09-14T05:36:45.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 374536192 bytes 2024-09-14T05:36:45.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 401601536 bytes 2024-09-14T05:36:46.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 441277440 bytes 2024-09-14T05:36:46.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 472003584 bytes 2024-09-14T05:36:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 478680064 bytes 2024-09-14T05:36:46.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 491373568 bytes 2024-09-14T05:36:46.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 563408896 bytes 2024-09-14T05:36:46.457 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 590407680 bytes 2024-09-14T05:36:46.538 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 621057024 bytes 2024-09-14T05:36:46.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 692085760 bytes 2024-09-14T05:36:46.798 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 761011200 bytes 2024-09-14T05:36:46.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 811788288 bytes 2024-09-14T05:36:47.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 916901888 bytes 2024-09-14T05:36:47.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 961458176 bytes 2024-09-14T05:36:47.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 962072576 bytes 2024-09-14T05:36:47.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1008090112 bytes 2024-09-14T05:36:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1023804416 bytes 2024-09-14T05:36:47.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1074378752 bytes 2024-09-14T05:36:47.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1160453120 bytes 2024-09-14T05:36:47.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1208148992 bytes 2024-09-14T05:36:47.815 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1247102976 bytes 2024-09-14T05:36:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1301296128 bytes 2024-09-14T05:36:48.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1344078848 bytes 2024-09-14T05:36:48.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1386384384 bytes 2024-09-14T05:36:48.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1417540608 bytes 2024-09-14T05:36:48.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1446786048 bytes 2024-09-14T05:36:48.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1468139520 bytes 2024-09-14T05:36:48.422 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1477389312 bytes 2024-09-14T05:36:48.489 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1489135616 bytes 2024-09-14T05:36:48.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1507979264 bytes 2024-09-14T05:36:48.654 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1541703680 bytes 2024-09-14T05:36:48.720 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1560332288 bytes 2024-09-14T05:36:48.852 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1647151104 bytes 2024-09-14T05:36:48.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1690793984 bytes 2024-09-14T05:36:49.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1712056320 bytes 2024-09-14T05:36:49.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 1741361152 bytes 2024-09-14T05:36:49.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 1815783424 bytes 2024-09-14T05:36:49.312 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 1851272192 bytes 2024-09-14T05:36:49.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 1897442304 bytes 2024-09-14T05:36:49.456 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 1898037248 bytes 2024-09-14T05:36:49.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 1908008960 bytes 2024-09-14T05:36:49.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:36:49.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:36:49.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:36:49.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:36:49.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-14T05:36:49.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:36:49.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:36:49.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:36:49.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-14T05:36:49.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local description_pattern= 2024-09-14T05:36:49.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local service_pattern= 2024-09-14T05:36:49.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:36:49.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:36:50.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-14T05:36:50.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:36:50.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:36:50.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:36:50.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-14T05:36:50.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local description_pattern= 2024-09-14T05:36:50.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local service_pattern= 2024-09-14T05:36:50.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status 2024-09-14T05:36:50.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CEPH_ARGS= 2024-09-14T05:36:50.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-14T05:36:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status='test: 2024-09-14T05:36:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: global_id: 77511074-da27-44b2-8b96-37f8b827a31e 2024-09-14T05:36:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: state: up+replaying 2024-09-14T05:36:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":268289536.0,"last_snapshot_bytes":536579072,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726292148,"remote_snapshot_timestamp":1726292148,"replay_state":"idle"} 2024-09-14T05:36:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: service: mirror.2 on smithi191 2024-09-14T05:36:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: last_update: 2024-09-14 05:36:35 2024-09-14T05:36:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: peer_sites: 2024-09-14T05:36:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: name: cluster2 2024-09-14T05:36:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: state: up+stopped 2024-09-14T05:36:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: description: local image is primary 2024-09-14T05:36:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: last_update: 2024-09-14 05:36:35' 2024-09-14T05:36:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^ state: .*up+replaying' 2024-09-14T05:36:50.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: state: up+replaying 2024-09-14T05:36:50.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^ description: .*' 2024-09-14T05:36:50.636 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":268289536.0,"last_snapshot_bytes":536579072,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726292148,"remote_snapshot_timestamp":1726292148,"replay_state":"idle"} 2024-09-14T05:36:50.636 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:36:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo up+replaying 2024-09-14T05:36:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^up+' 2024-09-14T05:36:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:up+replaying 2024-09-14T05:36:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep 'service: *mirror..* on ' 2024-09-14T05:36:50.638 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: service: mirror.2 on smithi191 2024-09-14T05:36:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local last_update 2024-09-14T05:36:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-14T05:36:50.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ last_update='2024-09-14 05:36:35' 2024-09-14T05:36:50.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-14 05:36:35' 2024-09-14T05:36:50.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:36:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:36:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:36:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-14T05:36:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local 'prev_last_update=2024-09-14 05:36:35' 2024-09-14T05:36:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status 2024-09-14T05:36:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CEPH_ARGS= 2024-09-14T05:36:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-14T05:36:50.703 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status='OKOKOK142164296mirror.1smithi19119.3.0-4775-g7f24e589falseOK42204289mirror.2smithi19119.3.0-4775-g7f24e589trueOK42244292mirror.3smithi19119.3.0-4775-g7f24e589falseOK43704372mirror.0smithi19119.3.0-4775-g7f24e589falseOKtest77511074-da27-44b2-8b96-37f8b827a31eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":268289536.0,"last_snapshot_bytes":536579072,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726292148,"remote_snapshot_timestamp":1726292148,"replay_state":"idle"}42204289mirror.2smithi1912024-09-14 05:36:35cluster29d38de88-cffd-4132-b7e8-809c69f2745aup+stoppedlocal image is primary2024-09-14 05:36:35' 2024-09-14T05:36:50.703 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local last_update state 2024-09-14T05:36:50.703 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-14T05:36:50.704 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ last_update='2024-09-14 05:36:35' 2024-09-14T05:36:50.705 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-14T05:36:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ state=up+replaying 2024-09-14T05:36:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo up+replaying 2024-09-14T05:36:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep up+replaying 2024-09-14T05:36:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:up+replaying 2024-09-14T05:36:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:36:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:36:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap3 2024-09-14T05:36:50.709 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror test snap3 2024-09-14T05:36:50.709 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:36:50.709 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:36:50.709 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:36:50.709 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap3 2024-09-14T05:36:50.709 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:37:21.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-14T05:37:21.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:37:21.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:37:21.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:37:21.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-14T05:37:21.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:37:21.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:37:21.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:37:21.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:37:21.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:37:21.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:37:21.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:37:22.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-14T05:37:22.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:37:22.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:37:22.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:37:22.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:37:22.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:37:22.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:37:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-14T05:37:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:37:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:37:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:37:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:37:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:37:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:37:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:37:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:37:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:37:22.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:37:22.014 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:37:22.014 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-14T05:37:22.014 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:37:22.014 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:37:22.014 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:37:22.014 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:37:22.014 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:37:22.014 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:37:22.014 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:37:22.014 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:37:22.014 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:37:22.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:37:22.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:37:22.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:37:22.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:37:22.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:37:22.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:37:22.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:37:22.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:37:22.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:37:22.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:37:22.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:37:22.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:37:22.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:37:22.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:37:22.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:37:22.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:37:22.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:37:22.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:37:22.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:37:22.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:37:22.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:37:22.037 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:37:22.037 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:37:22.037 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:37:22.037 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-14T05:37:22.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T05:37:22.117 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:37:22.117 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:37:22.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:37:22.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-14T05:37:22.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-14T05:37:22.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:37:22.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:37:22.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T05:37:22.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-14T05:37:22.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:37:22.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:37:22.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:37:22.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-14T05:37:22.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T05:37:22.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:37:22.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:37:22.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-14T05:37:22.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:37:22.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T05:37:22.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-14T05:37:22.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:37:22.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:37:22.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T05:37:22.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-14T05:37:22.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:37:22.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:37:22.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:37:22.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:37:22.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T05:37:22.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:37:22.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:37:22.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-14T05:37:22.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-14T05:37:22.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T05:37:22.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:37:22.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:37:22.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:37:22.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-14T05:37:22.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T05:37:22.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:37:22.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:37:22.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:37:22.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-14T05:37:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-14T05:37:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:37:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:37:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T05:37:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-14T05:37:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:37:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:37:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:37:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-14T05:37:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T05:37:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:37:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:37:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-14T05:37:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:37:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T05:37:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-14T05:37:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:37:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:37:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T05:37:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-14T05:37:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:37:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:37:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:37:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:37:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T05:37:22.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:37:22.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:37:22.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-14T05:37:22.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-14T05:37:22.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T05:37:22.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:37:22.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:37:22.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:37:22.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-14T05:37:22.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:37:22.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:37:22.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:37:22.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:37:22.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:37:22.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:37:22.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:37:22.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:37:22.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-14T05:37:22.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:37:22.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:37:22.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:37:22.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:37:22.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:37:22.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:37:22.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-14T05:37:22.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:37:22.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:37:22.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:37:22.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:37:22.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-test.status 2024-09-14T05:37:22.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-test.status 2024-09-14T05:37:22.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:37:22.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.status 2024-09-14T05:37:22.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-test.status 2024-09-14T05:37:22.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-test.status 2024-09-14T05:37:22.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:37:22.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:37:22.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-14T05:37:22.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:37:22.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:37:22.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:37:22.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-14T05:37:24.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 11 2024-09-14T05:37:24.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.status 2024-09-14T05:37:24.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:37:24.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:37:24.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:37:24.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.status 2024-09-14T05:37:24.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-14T05:37:24.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:37:24.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:37:24.101 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=11 2024-09-14T05:37:24.101 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:37:24.101 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:37:24.101 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:37:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:37:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:37:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:37:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:37:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:37:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T05:37:24.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:37:24.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:37:24.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:37:24.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:37:24.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:37:24.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:37:24.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:37:24.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 11 = '' 2024-09-14T05:37:24.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:37:24.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:37:24.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:37:24.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:37:24.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:37:24.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:37:24.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:37:24.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T05:37:24.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:37:24.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:37:24.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:37:24.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:37:24.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:37:24.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:37:24.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:37:24.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 11 = '' 2024-09-14T05:37:24.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:37:24.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:37:25.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:37:25.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:37:25.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:37:25.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:37:25.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:37:25.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T05:37:25.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:37:25.660 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:37:25.660 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:37:25.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:37:25.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:37:25.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:37:25.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:37:25.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 11 = '' 2024-09-14T05:37:25.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:37:25.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-14T05:37:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:37:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:37:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:37:27.264 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:37:27.264 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:37:27.264 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T05:37:27.264 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:37:27.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:37:27.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=11 2024-09-14T05:37:27.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 11 = 11 2024-09-14T05:37:27.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:37:27.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror test snap3 2024-09-14T05:37:27.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:37:27.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:37:27.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:37:27.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap3 2024-09-14T05:37:27.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:37:27.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:37:27.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:37:28.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/test@snap3 2024-09-14T05:37:28.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'test': 2024-09-14T05:37:28.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 512 MiB in 128 objects 2024-09-14T05:37:28.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:37:28.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 6 2024-09-14T05:37:28.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c1a41659ac 2024-09-14T05:37:28.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c1a41659ac 2024-09-14T05:37:28.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:37:28.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:37:28.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:37:28.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:37:28.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:35:08 2024 2024-09-14T05:37:28.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:36:41 2024 2024-09-14T05:37:28.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:35:08 2024 2024-09-14T05:37:28.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:37:28.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:37:28.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:37:28.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 77511074-da27-44b2-8b96-37f8b827a31e 2024-09-14T05:37:28.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:37:28.369 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:37:28.369 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snap2-clean ']' 2024-09-14T05:37:28.369 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap2-clean 2024-09-14T05:37:28.369 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:37:28.369 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:37:28.369 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap2-clean 2024-09-14T05:37:28.369 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:37:28.369 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:37:28.370 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:37:28.370 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:37:28.370 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:37:28.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:37:28.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:37:29.657 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:37:29.668 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:37:29.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:37:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:37:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap3 2024-09-14T05:37:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:37:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:37:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap3 2024-09-14T05:37:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:37:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:37:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:37:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:37:29.995 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:37:30.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:37:30.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:37:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:37:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:37:31.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:37:31.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:37:31.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ clean_snap_name=snap3-clean 2024-09-14T05:37:31.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror test snap3-clean 2024-09-14T05:37:31.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:37:31.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:37:31.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:37:31.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap3-clean 2024-09-14T05:37:31.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3-clean --debug-rbd=20 --debug-journaler=20 2024-09-14T05:37:32.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-14T05:37:32.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror test 2024-09-14T05:37:32.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:37:32.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:37:32.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:37:32.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:37:32.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=7 2024-09-14T05:37:32.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:37:32.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-14T05:37:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 59563008 bytes 2024-09-14T05:37:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 179802112 bytes 2024-09-14T05:37:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 227267584 bytes 2024-09-14T05:37:33.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 301867008 bytes 2024-09-14T05:37:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 339000320 bytes 2024-09-14T05:37:33.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 386742272 bytes 2024-09-14T05:37:33.577 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 425619456 bytes 2024-09-14T05:37:33.696 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 521519104 bytes 2024-09-14T05:37:33.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 586345472 bytes 2024-09-14T05:37:33.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 628512768 bytes 2024-09-14T05:37:34.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 637062144 bytes 2024-09-14T05:37:34.113 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 698256384 bytes 2024-09-14T05:37:34.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 717790208 bytes 2024-09-14T05:37:34.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 748421120 bytes 2024-09-14T05:37:34.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 784573440 bytes 2024-09-14T05:37:34.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 841776128 bytes 2024-09-14T05:37:34.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 871610368 bytes 2024-09-14T05:37:34.683 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 932951040 bytes 2024-09-14T05:37:34.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 956805120 bytes 2024-09-14T05:37:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 1005418496 bytes 2024-09-14T05:37:34.966 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 1052996608 bytes 2024-09-14T05:37:35.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 1065701376 bytes 2024-09-14T05:37:35.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1114917888 bytes 2024-09-14T05:37:35.228 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1166932992 bytes 2024-09-14T05:37:35.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1193730048 bytes 2024-09-14T05:37:35.421 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1232899072 bytes 2024-09-14T05:37:35.534 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1294629888 bytes 2024-09-14T05:37:35.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1341127680 bytes 2024-09-14T05:37:35.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1369978880 bytes 2024-09-14T05:37:35.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1400956928 bytes 2024-09-14T05:37:35.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1459780608 bytes 2024-09-14T05:37:36.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1507019776 bytes 2024-09-14T05:37:36.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1544862720 bytes 2024-09-14T05:37:36.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1560344576 bytes 2024-09-14T05:37:36.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1611987968 bytes 2024-09-14T05:37:36.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1622585344 bytes 2024-09-14T05:37:36.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1656572928 bytes 2024-09-14T05:37:36.596 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1744984064 bytes 2024-09-14T05:37:36.731 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1814053888 bytes 2024-09-14T05:37:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1880930304 bytes 2024-09-14T05:37:36.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1931916288 bytes 2024-09-14T05:37:37.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1955419136 bytes 2024-09-14T05:37:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1978798080 bytes 2024-09-14T05:37:37.301 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 2090040320 bytes 2024-09-14T05:37:37.399 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 2145806336 bytes 2024-09-14T05:37:37.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 2207588352 bytes 2024-09-14T05:37:37.643 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 2274419712 bytes 2024-09-14T05:37:37.731 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2306590720 bytes 2024-09-14T05:37:37.835 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2345974784 bytes 2024-09-14T05:37:37.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2388776960 bytes 2024-09-14T05:37:38.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2415230976 bytes 2024-09-14T05:37:38.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2423421952 bytes 2024-09-14T05:37:38.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2474565632 bytes 2024-09-14T05:37:38.328 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2556433408 bytes 2024-09-14T05:37:38.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2604656640 bytes 2024-09-14T05:37:38.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2659087360 bytes 2024-09-14T05:37:38.682 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2733080576 bytes 2024-09-14T05:37:38.795 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2778187776 bytes 2024-09-14T05:37:38.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2797983744 bytes 2024-09-14T05:37:38.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2829262848 bytes 2024-09-14T05:37:39.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2852842496 bytes 2024-09-14T05:37:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2904626176 bytes 2024-09-14T05:37:39.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2961399808 bytes 2024-09-14T05:37:39.352 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2982350848 bytes 2024-09-14T05:37:39.470 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 3051774976 bytes 2024-09-14T05:37:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:37:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:37:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:37:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:37:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-14T05:37:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:37:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:37:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:37:39.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-14T05:37:39.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local description_pattern= 2024-09-14T05:37:39.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local service_pattern= 2024-09-14T05:37:39.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:37:39.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:37:40.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-14T05:37:40.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:37:40.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:37:40.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:37:40.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-14T05:37:40.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local description_pattern= 2024-09-14T05:37:40.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local service_pattern= 2024-09-14T05:37:40.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status 2024-09-14T05:37:40.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CEPH_ARGS= 2024-09-14T05:37:40.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-14T05:37:40.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status='test: 2024-09-14T05:37:40.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: global_id: 77511074-da27-44b2-8b96-37f8b827a31e 2024-09-14T05:37:40.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: state: up+replaying 2024-09-14T05:37:40.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: description: replaying, {"bytes_per_second":17272797.87,"bytes_per_snapshot":526627840.0,"last_snapshot_bytes":518183936,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726292242,"remote_snapshot_timestamp":1726292242,"replay_state":"idle"} 2024-09-14T05:37:40.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: service: mirror.2 on smithi191 2024-09-14T05:37:40.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: last_update: 2024-09-14 05:37:35 2024-09-14T05:37:40.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: peer_sites: 2024-09-14T05:37:40.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: name: cluster2 2024-09-14T05:37:40.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: state: up+stopped 2024-09-14T05:37:40.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: description: local image is primary 2024-09-14T05:37:40.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: last_update: 2024-09-14 05:37:35' 2024-09-14T05:37:40.570 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^ state: .*up+replaying' 2024-09-14T05:37:40.570 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: state: up+replaying 2024-09-14T05:37:40.570 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^ description: .*' 2024-09-14T05:37:40.571 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: description: replaying, {"bytes_per_second":17272797.87,"bytes_per_snapshot":526627840.0,"last_snapshot_bytes":518183936,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726292242,"remote_snapshot_timestamp":1726292242,"replay_state":"idle"} 2024-09-14T05:37:40.571 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:37:40.571 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo up+replaying 2024-09-14T05:37:40.571 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^up+' 2024-09-14T05:37:40.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:up+replaying 2024-09-14T05:37:40.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep 'service: *mirror..* on ' 2024-09-14T05:37:40.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: service: mirror.2 on smithi191 2024-09-14T05:37:40.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local last_update 2024-09-14T05:37:40.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-14T05:37:40.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ last_update='2024-09-14 05:37:35' 2024-09-14T05:37:40.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-14 05:37:35' 2024-09-14T05:37:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:37:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:37:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:37:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-14T05:37:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local 'prev_last_update=2024-09-14 05:37:35' 2024-09-14T05:37:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status 2024-09-14T05:37:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CEPH_ARGS= 2024-09-14T05:37:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-14T05:37:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status='OKOKOK142164296mirror.1smithi19119.3.0-4775-g7f24e589falseOK42204289mirror.2smithi19119.3.0-4775-g7f24e589trueOK42244292mirror.3smithi19119.3.0-4775-g7f24e589falseOK43704372mirror.0smithi19119.3.0-4775-g7f24e589falseOKtest77511074-da27-44b2-8b96-37f8b827a31eup+replayingreplaying, {"bytes_per_second":17272797.87,"bytes_per_snapshot":526627840.0,"last_snapshot_bytes":518183936,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726292242,"remote_snapshot_timestamp":1726292242,"replay_state":"idle"}42204289mirror.2smithi1912024-09-14 05:37:35cluster29d38de88-cffd-4132-b7e8-809c69f2745aup+stoppedlocal image is primary2024-09-14 05:37:35' 2024-09-14T05:37:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local last_update state 2024-09-14T05:37:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-14T05:37:40.639 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ last_update='2024-09-14 05:37:35' 2024-09-14T05:37:40.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-14T05:37:40.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ state=up+replaying 2024-09-14T05:37:40.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo up+replaying 2024-09-14T05:37:40.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep up+replaying 2024-09-14T05:37:40.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:up+replaying 2024-09-14T05:37:40.643 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:37:40.643 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:37:40.643 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap4 2024-09-14T05:37:40.643 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror test snap4 2024-09-14T05:37:40.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:37:40.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:37:40.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:37:40.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap4 2024-09-14T05:37:40.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:37:51.438 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9358256934409512) > weight (0.25) 2024-09-14T05:37:51.439 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7527055528412282) > weight (0.25) 2024-09-14T05:37:51.439 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.6694608940756885) > weight (0.25) 2024-09-14T05:37:51.439 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2024-09-14T05:37:51.439 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2024-09-14T05:37:51.439 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-09-14T05:37:51.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-14T05:37:51.439+0000 7f7d66132640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 36820) UID: 1000 2024-09-14T05:37:54.440 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2024-09-14T05:37:54.441 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T05:37:54.441 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2024-09-14T05:37:54.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-09-14T05:37:54.441 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-09-14T05:37:54.444 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-09-14T05:37:54.445 INFO:tasks.rbd_mirror_thrash:waiting for 108 secs before thrashing 2024-09-14T05:38:09.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-14T05:38:09.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:38:09.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:38:09.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:38:09.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-14T05:38:09.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:38:09.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:38:09.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:38:09.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:38:09.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:38:09.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:38:09.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:38:10.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-14T05:38:10.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:38:10.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:38:10.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:38:10.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:38:10.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:38:10.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:38:10.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-14T05:38:10.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:38:10.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:38:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:38:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:38:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:38:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:38:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:38:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:38:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:38:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:38:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:38:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-14T05:38:10.036 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:38:10.036 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:38:10.036 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:38:10.036 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:38:10.036 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:38:10.036 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:38:10.036 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:38:10.036 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:38:10.036 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:38:10.036 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:38:10.036 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:38:10.036 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:38:10.036 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:38:10.037 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:38:10.037 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:38:10.037 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:38:10.037 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:38:10.037 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:38:10.037 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:38:10.037 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:38:10.037 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:38:10.037 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:38:10.037 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:38:10.037 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:38:10.037 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:38:10.038 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:38:10.038 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:38:10.038 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:38:10.038 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:38:10.038 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:38:10.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:38:10.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:38:10.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:38:10.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-14T05:38:10.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T05:38:10.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:38:10.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:38:10.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:38:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-14T05:38:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-14T05:38:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:38:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:38:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T05:38:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-14T05:38:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:38:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:38:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:38:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-14T05:38:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T05:38:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:38:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:38:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-14T05:38:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:38:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T05:38:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-14T05:38:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:38:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:38:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T05:38:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-14T05:38:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:38:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:38:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:38:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:38:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T05:38:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:38:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:38:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-14T05:38:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-14T05:38:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T05:38:10.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:38:10.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:38:10.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:38:10.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-14T05:38:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T05:38:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:38:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:38:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:38:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-14T05:38:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-14T05:38:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:38:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:38:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T05:38:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-14T05:38:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:38:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:38:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:38:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-14T05:38:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T05:38:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:38:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:38:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-14T05:38:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:38:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T05:38:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-14T05:38:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:38:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:38:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T05:38:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-14T05:38:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:38:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:38:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:38:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:38:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T05:38:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:38:10.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:38:10.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-14T05:38:10.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-14T05:38:10.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T05:38:10.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:38:10.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:38:10.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:38:10.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-14T05:38:10.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:38:10.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:38:10.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:38:10.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:38:10.340 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:38:10.341 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:38:10.341 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:38:10.341 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:38:10.341 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-14T05:38:10.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:38:10.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:38:10.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:38:10.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:38:10.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:38:10.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:38:10.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-14T05:38:10.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:38:10.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:38:10.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:38:10.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:38:10.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-test.status 2024-09-14T05:38:10.343 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-test.status 2024-09-14T05:38:10.343 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:38:10.343 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.status 2024-09-14T05:38:10.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-test.status 2024-09-14T05:38:10.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-test.status 2024-09-14T05:38:10.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:38:10.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:38:10.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-14T05:38:10.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:38:10.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:38:10.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:38:10.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-14T05:38:12.021 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 14 2024-09-14T05:38:12.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.status 2024-09-14T05:38:12.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:38:12.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:38:12.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:38:12.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.status 2024-09-14T05:38:12.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-14T05:38:12.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:38:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:38:12.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=14 2024-09-14T05:38:12.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:38:12.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:38:12.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:38:12.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:38:12.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:38:12.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:38:12.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:38:12.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:38:12.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T05:38:12.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:38:12.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:38:12.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:38:12.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:38:12.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:38:12.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:38:12.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:38:12.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 14 = '' 2024-09-14T05:38:12.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:38:12.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:38:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:38:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:38:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:38:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:38:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:38:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T05:38:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:38:12.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:38:12.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:38:12.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:38:12.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:38:12.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:38:12.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:38:12.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 14 = '' 2024-09-14T05:38:12.821 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:38:12.821 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:38:13.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:38:13.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:38:13.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:38:13.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:38:13.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:38:13.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T05:38:13.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:38:13.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:38:13.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:38:13.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:38:13.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:38:13.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:38:13.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:38:13.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 14 = '' 2024-09-14T05:38:13.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:38:13.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-14T05:38:15.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:38:15.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:38:15.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:38:15.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:38:15.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:38:15.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T05:38:15.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:38:15.329 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:38:15.331 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=14 2024-09-14T05:38:15.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 14 = 14 2024-09-14T05:38:15.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:38:15.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror test snap4 2024-09-14T05:38:15.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:38:15.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:38:15.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:38:15.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap4 2024-09-14T05:38:15.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:38:15.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:38:15.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:38:16.333 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/test@snap4 2024-09-14T05:38:16.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'test': 2024-09-14T05:38:16.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 512 MiB in 128 objects 2024-09-14T05:38:16.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:38:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 8 2024-09-14T05:38:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c1a41659ac 2024-09-14T05:38:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c1a41659ac 2024-09-14T05:38:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:38:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:38:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:38:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:38:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:35:08 2024 2024-09-14T05:38:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:36:41 2024 2024-09-14T05:38:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:35:08 2024 2024-09-14T05:38:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:38:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:38:16.387 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:38:16.387 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 77511074-da27-44b2-8b96-37f8b827a31e 2024-09-14T05:38:16.387 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:38:16.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:38:16.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snap3-clean ']' 2024-09-14T05:38:16.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap3-clean 2024-09-14T05:38:16.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:38:16.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:38:16.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap3-clean 2024-09-14T05:38:16.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:38:16.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:38:16.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:38:16.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:38:16.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:38:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:38:16.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:38:17.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:38:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:38:17.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:38:17.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:38:17.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap4 2024-09-14T05:38:17.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:38:17.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:38:17.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap4 2024-09-14T05:38:17.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:38:17.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:38:17.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:38:17.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:38:17.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:38:18.459 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:38:18.470 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:38:19.117 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:38:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:38:19.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:38:19.453 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:38:19.453 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ clean_snap_name=snap4-clean 2024-09-14T05:38:19.453 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror test snap4-clean 2024-09-14T05:38:19.453 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:38:19.453 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:38:19.453 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:38:19.454 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap4-clean 2024-09-14T05:38:19.454 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4-clean --debug-rbd=20 --debug-journaler=20 2024-09-14T05:38:19.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-14T05:38:19.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror test 2024-09-14T05:38:19.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:38:19.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:38:19.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:38:19.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:38:19.983 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=13 2024-09-14T05:38:19.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:38:19.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-14T05:38:20.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 27570176 bytes 2024-09-14T05:38:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 67989504 bytes 2024-09-14T05:38:20.478 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 102369280 bytes 2024-09-14T05:38:20.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 141188096 bytes 2024-09-14T05:38:20.731 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 242787328 bytes 2024-09-14T05:38:20.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 288934912 bytes 2024-09-14T05:38:20.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 305726464 bytes 2024-09-14T05:38:21.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 336790528 bytes 2024-09-14T05:38:21.112 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 378366976 bytes 2024-09-14T05:38:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 425755648 bytes 2024-09-14T05:38:21.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 429484032 bytes 2024-09-14T05:38:21.381 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 472684544 bytes 2024-09-14T05:38:21.502 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 553710592 bytes 2024-09-14T05:38:21.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 633235456 bytes 2024-09-14T05:38:21.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 661502976 bytes 2024-09-14T05:38:21.811 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 685440000 bytes 2024-09-14T05:38:21.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 724037632 bytes 2024-09-14T05:38:22.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 793878528 bytes 2024-09-14T05:38:22.091 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 796913664 bytes 2024-09-14T05:38:22.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 853554176 bytes 2024-09-14T05:38:22.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 902991872 bytes 2024-09-14T05:38:22.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 981222400 bytes 2024-09-14T05:38:22.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1110715392 bytes 2024-09-14T05:38:22.698 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1115259904 bytes 2024-09-14T05:38:22.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1117396992 bytes 2024-09-14T05:38:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1135883264 bytes 2024-09-14T05:38:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1152210944 bytes 2024-09-14T05:38:23.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1178592256 bytes 2024-09-14T05:38:23.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1216921600 bytes 2024-09-14T05:38:23.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1242659840 bytes 2024-09-14T05:38:23.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1316109312 bytes 2024-09-14T05:38:23.441 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1345130496 bytes 2024-09-14T05:38:23.550 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1409780736 bytes 2024-09-14T05:38:23.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1451801600 bytes 2024-09-14T05:38:23.736 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1472134144 bytes 2024-09-14T05:38:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1534004224 bytes 2024-09-14T05:38:23.947 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1577789440 bytes 2024-09-14T05:38:24.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1617961984 bytes 2024-09-14T05:38:24.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1699357696 bytes 2024-09-14T05:38:24.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1716788224 bytes 2024-09-14T05:38:24.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1758352384 bytes 2024-09-14T05:38:24.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1800922112 bytes 2024-09-14T05:38:24.528 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1808525312 bytes 2024-09-14T05:38:24.612 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1831641088 bytes 2024-09-14T05:38:24.737 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1893056512 bytes 2024-09-14T05:38:24.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1929535488 bytes 2024-09-14T05:38:24.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 1955590144 bytes 2024-09-14T05:38:25.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2018878464 bytes 2024-09-14T05:38:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2059594752 bytes 2024-09-14T05:38:25.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2128166912 bytes 2024-09-14T05:38:25.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2156331008 bytes 2024-09-14T05:38:25.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2194419712 bytes 2024-09-14T05:38:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2217632768 bytes 2024-09-14T05:38:25.626 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2260338688 bytes 2024-09-14T05:38:25.719 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2291957760 bytes 2024-09-14T05:38:25.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2320106496 bytes 2024-09-14T05:38:25.899 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2379299840 bytes 2024-09-14T05:38:25.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2381796352 bytes 2024-09-14T05:38:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2455934976 bytes 2024-09-14T05:38:26.247 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2525119488 bytes 2024-09-14T05:38:26.373 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2601176064 bytes 2024-09-14T05:38:26.613 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2630206464 bytes 2024-09-14T05:38:26.700 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2654836736 bytes 2024-09-14T05:38:26.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2714554368 bytes 2024-09-14T05:38:26.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2731875328 bytes 2024-09-14T05:38:26.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 2751226880 bytes 2024-09-14T05:38:27.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 2800913408 bytes 2024-09-14T05:38:27.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 2837431296 bytes 2024-09-14T05:38:27.299 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 2900320256 bytes 2024-09-14T05:38:27.384 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 2932555776 bytes 2024-09-14T05:38:27.452 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 2933171200 bytes 2024-09-14T05:38:27.548 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 2966167552 bytes 2024-09-14T05:38:27.623 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7300 IOs, 2980582400 bytes 2024-09-14T05:38:27.689 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7400 IOs, 2981214208 bytes 2024-09-14T05:38:27.795 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7500 IOs, 3030346752 bytes 2024-09-14T05:38:27.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7600 IOs, 3053030400 bytes 2024-09-14T05:38:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7700 IOs, 3072235520 bytes 2024-09-14T05:38:28.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7800 IOs, 3132964864 bytes 2024-09-14T05:38:28.193 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7900 IOs, 3183526912 bytes 2024-09-14T05:38:28.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8000 IOs, 3286142976 bytes 2024-09-14T05:38:28.459 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8100 IOs, 3332743168 bytes 2024-09-14T05:38:28.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8200 IOs, 3388718080 bytes 2024-09-14T05:38:28.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8300 IOs, 3411218432 bytes 2024-09-14T05:38:28.779 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8400 IOs, 3460856832 bytes 2024-09-14T05:38:28.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8500 IOs, 3504167936 bytes 2024-09-14T05:38:28.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8600 IOs, 3525837824 bytes 2024-09-14T05:38:29.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8700 IOs, 3577664512 bytes 2024-09-14T05:38:29.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8800 IOs, 3605838848 bytes 2024-09-14T05:38:29.247 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8900 IOs, 3630438400 bytes 2024-09-14T05:38:29.352 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9000 IOs, 3682525184 bytes 2024-09-14T05:38:29.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9100 IOs, 3757967360 bytes 2024-09-14T05:38:29.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9200 IOs, 3802560512 bytes 2024-09-14T05:38:29.669 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9300 IOs, 3826561024 bytes 2024-09-14T05:38:29.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9400 IOs, 3850533888 bytes 2024-09-14T05:38:29.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9500 IOs, 3854675968 bytes 2024-09-14T05:38:29.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9600 IOs, 3921177600 bytes 2024-09-14T05:38:30.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9700 IOs, 3965385728 bytes 2024-09-14T05:38:30.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9800 IOs, 4021571584 bytes 2024-09-14T05:38:30.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9900 IOs, 4081403904 bytes 2024-09-14T05:38:30.396 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10000 IOs, 4140465152 bytes 2024-09-14T05:38:30.444 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3354748341226349) > weight (0.25) 2024-09-14T05:38:30.444 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.6524718244038713) > weight (0.25) 2024-09-14T05:38:30.444 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.4711789090028178) > weight (0.25) 2024-09-14T05:38:30.444 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.26696443855951013) > weight (0.25) 2024-09-14T05:38:30.444 INFO:tasks.rbd_mirror_thrash:waiting for 85 secs before thrashing 2024-09-14T05:38:30.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10100 IOs, 4164980736 bytes 2024-09-14T05:38:30.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10200 IOs, 4183770112 bytes 2024-09-14T05:38:30.697 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10300 IOs, 4267373568 bytes 2024-09-14T05:38:30.791 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10400 IOs, 4295332864 bytes 2024-09-14T05:38:30.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10500 IOs, 4380242944 bytes 2024-09-14T05:38:31.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10600 IOs, 4441803776 bytes 2024-09-14T05:38:31.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10700 IOs, 4477007872 bytes 2024-09-14T05:38:31.226 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10800 IOs, 4477603840 bytes 2024-09-14T05:38:31.299 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10900 IOs, 4486194176 bytes 2024-09-14T05:38:31.375 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:11000 IOs, 4502755328 bytes 2024-09-14T05:38:31.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:11100 IOs, 4561057792 bytes 2024-09-14T05:38:31.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:11200 IOs, 4578477056 bytes 2024-09-14T05:38:31.684 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:11300 IOs, 4642644992 bytes 2024-09-14T05:38:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:11400 IOs, 4758528000 bytes 2024-09-14T05:38:31.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:11500 IOs, 4782086144 bytes 2024-09-14T05:38:32.069 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:11600 IOs, 4848877568 bytes 2024-09-14T05:38:32.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:11700 IOs, 4849482752 bytes 2024-09-14T05:38:32.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:11800 IOs, 4890386432 bytes 2024-09-14T05:38:32.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:11900 IOs, 4942606336 bytes 2024-09-14T05:38:32.455 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:12000 IOs, 4982350848 bytes 2024-09-14T05:38:32.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:12100 IOs, 5018650624 bytes 2024-09-14T05:38:32.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:12200 IOs, 5021211648 bytes 2024-09-14T05:38:32.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:12300 IOs, 5055506432 bytes 2024-09-14T05:38:32.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:38:32.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:38:32.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:38:32.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:38:32.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-14T05:38:32.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:38:32.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:38:32.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:38:32.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-14T05:38:32.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local description_pattern= 2024-09-14T05:38:32.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local service_pattern= 2024-09-14T05:38:32.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:38:32.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:38:33.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-14T05:38:33.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:38:33.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:38:33.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:38:33.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-14T05:38:33.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local description_pattern= 2024-09-14T05:38:33.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local service_pattern= 2024-09-14T05:38:33.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status 2024-09-14T05:38:33.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CEPH_ARGS= 2024-09-14T05:38:33.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-14T05:38:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status='test: 2024-09-14T05:38:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: global_id: 77511074-da27-44b2-8b96-37f8b827a31e 2024-09-14T05:38:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: state: up+replaying 2024-09-14T05:38:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":526627840.0,"last_snapshot_bytes":518183936,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726292242,"remote_snapshot_timestamp":1726292242,"replay_state":"idle"} 2024-09-14T05:38:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: service: mirror.2 on smithi191 2024-09-14T05:38:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: last_update: 2024-09-14 05:38:05 2024-09-14T05:38:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: peer_sites: 2024-09-14T05:38:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: name: cluster2 2024-09-14T05:38:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: state: up+stopped 2024-09-14T05:38:34.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: description: local image is primary 2024-09-14T05:38:34.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: last_update: 2024-09-14 05:38:05' 2024-09-14T05:38:34.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^ state: .*up+replaying' 2024-09-14T05:38:34.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: state: up+replaying 2024-09-14T05:38:34.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^ description: .*' 2024-09-14T05:38:34.051 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":526627840.0,"last_snapshot_bytes":518183936,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726292242,"remote_snapshot_timestamp":1726292242,"replay_state":"idle"} 2024-09-14T05:38:34.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:38:34.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo up+replaying 2024-09-14T05:38:34.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^up+' 2024-09-14T05:38:34.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:up+replaying 2024-09-14T05:38:34.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep 'service: *mirror..* on ' 2024-09-14T05:38:34.054 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: service: mirror.2 on smithi191 2024-09-14T05:38:34.054 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local last_update 2024-09-14T05:38:34.054 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-14T05:38:34.056 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ last_update='2024-09-14 05:38:05' 2024-09-14T05:38:34.056 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-14 05:38:05' 2024-09-14T05:38:34.056 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:38:34.056 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:38:34.056 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:38:34.056 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-14T05:38:34.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local 'prev_last_update=2024-09-14 05:38:05' 2024-09-14T05:38:34.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status 2024-09-14T05:38:34.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CEPH_ARGS= 2024-09-14T05:38:34.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-14T05:38:34.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status='OKOKOK142164296mirror.1smithi19119.3.0-4775-g7f24e589falseOK42204289mirror.2smithi19119.3.0-4775-g7f24e589trueOK42244292mirror.3smithi19119.3.0-4775-g7f24e589falseOK43704372mirror.0smithi19119.3.0-4775-g7f24e589falseOKtest77511074-da27-44b2-8b96-37f8b827a31eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":526627840.0,"last_snapshot_bytes":518183936,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726292242,"remote_snapshot_timestamp":1726292242,"replay_state":"idle"}42204289mirror.2smithi1912024-09-14 05:38:05cluster29d38de88-cffd-4132-b7e8-809c69f2745aup+stoppedlocal image is primary2024-09-14 05:38:05' 2024-09-14T05:38:34.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local last_update state 2024-09-14T05:38:34.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-14T05:38:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ last_update='2024-09-14 05:38:05' 2024-09-14T05:38:34.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-14T05:38:34.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ state=up+replaying 2024-09-14T05:38:34.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo up+replaying 2024-09-14T05:38:34.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep up+replaying 2024-09-14T05:38:34.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:up+replaying 2024-09-14T05:38:34.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:38:34.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:38:34.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap5 2024-09-14T05:38:34.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror test snap5 2024-09-14T05:38:34.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:38:34.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:38:34.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:38:34.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap5 2024-09-14T05:38:34.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:39:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-14T05:39:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:39:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:39:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:39:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-14T05:39:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:39:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:39:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:39:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:39:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:39:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:39:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:39:05.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-14T05:39:05.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:39:05.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:39:05.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:39:05.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:39:05.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:39:05.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:39:05.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-14T05:39:05.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:39:05.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:39:05.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:39:05.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:39:05.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:39:05.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:39:05.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:39:05.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:39:05.542 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:39:05.542 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:39:05.542 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:39:05.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-14T05:39:05.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:39:05.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:39:05.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:39:05.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:39:05.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:39:05.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:39:05.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:39:05.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:39:05.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:39:05.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:39:05.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:39:05.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:39:05.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:39:05.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:39:05.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:39:05.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:39:05.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:39:05.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:39:05.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:39:05.545 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:39:05.545 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:39:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:39:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:39:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:39:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:39:05.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:39:05.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:39:05.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:39:05.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:39:05.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:39:05.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:39:05.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:39:05.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:39:05.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-14T05:39:05.645 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T05:39:05.645 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:39:05.646 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:39:05.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:39:05.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-14T05:39:05.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-14T05:39:05.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:39:05.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:39:05.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T05:39:05.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-14T05:39:05.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:39:05.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:39:05.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:39:05.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-14T05:39:05.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T05:39:05.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:39:05.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:39:05.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-14T05:39:05.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:39:05.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T05:39:05.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-14T05:39:05.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:39:05.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:39:05.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T05:39:05.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-14T05:39:05.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:39:05.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:39:05.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:39:05.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:39:05.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T05:39:05.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:39:05.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:39:05.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-14T05:39:05.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-14T05:39:05.652 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T05:39:05.669 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:39:05.669 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:39:05.669 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:39:05.669 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-14T05:39:05.744 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T05:39:05.744 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:39:05.744 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:39:05.747 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:39:05.748 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-14T05:39:05.748 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-14T05:39:05.748 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:39:05.748 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:39:05.748 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T05:39:05.748 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-14T05:39:05.748 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:39:05.748 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:39:05.748 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:39:05.748 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-14T05:39:05.748 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T05:39:05.749 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:39:05.749 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:39:05.749 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-14T05:39:05.749 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:39:05.749 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T05:39:05.749 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-14T05:39:05.749 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:39:05.749 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:39:05.749 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T05:39:05.749 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-14T05:39:05.749 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:39:05.749 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:39:05.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:39:05.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:39:05.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T05:39:05.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:39:05.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:39:05.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-14T05:39:05.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-14T05:39:05.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T05:39:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:39:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:39:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:39:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-14T05:39:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:39:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:39:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:39:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:39:05.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:39:05.851 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:39:05.852 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:39:05.852 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:39:05.852 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-14T05:39:05.852 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:39:05.852 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:39:05.852 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:39:05.852 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:39:05.852 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:39:05.852 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:39:05.852 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-14T05:39:05.852 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:39:05.852 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:39:05.852 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:39:05.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:39:05.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-test.status 2024-09-14T05:39:05.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-test.status 2024-09-14T05:39:05.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:39:05.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.status 2024-09-14T05:39:05.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-test.status 2024-09-14T05:39:05.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-test.status 2024-09-14T05:39:05.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:39:05.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:39:05.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-14T05:39:05.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:39:05.855 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:39:05.855 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:39:05.855 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-14T05:39:07.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 17 2024-09-14T05:39:07.564 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.status 2024-09-14T05:39:07.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:39:07.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:39:07.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:39:07.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.status 2024-09-14T05:39:07.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-14T05:39:07.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:39:07.637 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:39:07.639 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=17 2024-09-14T05:39:07.639 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:39:07.639 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:39:07.639 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:39:07.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:39:07.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:39:07.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:39:07.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:39:07.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:39:07.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T05:39:07.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:39:07.892 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:39:07.892 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:39:07.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:39:07.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:39:07.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:39:07.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:39:07.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 17 = '' 2024-09-14T05:39:07.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:39:07.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:39:08.295 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:39:08.295 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:39:08.295 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:39:08.295 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:39:08.295 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:39:08.295 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T05:39:08.295 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:39:08.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:39:08.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:39:08.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:39:08.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:39:08.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:39:08.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:39:08.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 17 = '' 2024-09-14T05:39:08.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:39:08.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:39:09.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:39:09.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:39:09.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:39:09.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:39:09.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:39:09.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T05:39:09.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:39:09.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:39:09.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:39:09.220 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:39:09.220 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:39:09.221 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:39:09.221 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:39:09.221 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 17 = '' 2024-09-14T05:39:09.221 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:39:09.221 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-14T05:39:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:39:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:39:10.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:39:10.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:39:10.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:39:10.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T05:39:10.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:39:10.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:39:10.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=17 2024-09-14T05:39:10.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 17 = 17 2024-09-14T05:39:10.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:39:10.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror test snap5 2024-09-14T05:39:10.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:39:10.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:39:10.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:39:10.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap5 2024-09-14T05:39:10.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:39:10.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:39:10.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:39:11.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/test@snap5 2024-09-14T05:39:11.932 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'test': 2024-09-14T05:39:11.932 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 512 MiB in 128 objects 2024-09-14T05:39:11.932 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:39:11.932 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 10 2024-09-14T05:39:11.932 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c1a41659ac 2024-09-14T05:39:11.932 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c1a41659ac 2024-09-14T05:39:11.932 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:39:11.932 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:39:11.932 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:39:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:39:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:35:08 2024 2024-09-14T05:39:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:38:16 2024 2024-09-14T05:39:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:35:08 2024 2024-09-14T05:39:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:39:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:39:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:39:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 77511074-da27-44b2-8b96-37f8b827a31e 2024-09-14T05:39:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:39:11.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:39:11.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snap4-clean ']' 2024-09-14T05:39:11.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap4-clean 2024-09-14T05:39:11.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:39:11.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:39:11.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap4-clean 2024-09-14T05:39:11.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:39:11.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:39:11.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:39:11.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:39:11.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:39:12.456 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:39:12.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:39:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:39:13.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:39:13.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:39:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:39:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap5 2024-09-14T05:39:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:39:13.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:39:13.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap5 2024-09-14T05:39:13.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:39:13.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:39:13.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:39:13.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:39:13.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:39:14.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:39:14.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:39:14.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:39:14.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:39:14.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:39:15.084 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:39:15.085 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ clean_snap_name=snap5-clean 2024-09-14T05:39:15.085 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror test snap5-clean 2024-09-14T05:39:15.085 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:39:15.085 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:39:15.085 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:39:15.085 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap5-clean 2024-09-14T05:39:15.085 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5-clean --debug-rbd=20 --debug-journaler=20 2024-09-14T05:39:15.563 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-14T05:39:15.563 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror test 2024-09-14T05:39:15.563 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:39:15.564 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:39:15.564 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:39:15.564 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:39:15.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=11 2024-09-14T05:39:15.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:39:15.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-14T05:39:15.863 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 80154624 bytes 2024-09-14T05:39:16.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 129291264 bytes 2024-09-14T05:39:16.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 170885120 bytes 2024-09-14T05:39:16.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 193941504 bytes 2024-09-14T05:39:16.331 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 234106880 bytes 2024-09-14T05:39:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 306782208 bytes 2024-09-14T05:39:16.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 337711104 bytes 2024-09-14T05:39:16.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 357843968 bytes 2024-09-14T05:39:16.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 392331264 bytes 2024-09-14T05:39:16.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 462963712 bytes 2024-09-14T05:39:17.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 544814080 bytes 2024-09-14T05:39:17.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 580678656 bytes 2024-09-14T05:39:17.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 617433088 bytes 2024-09-14T05:39:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 657136640 bytes 2024-09-14T05:39:17.457 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 713259008 bytes 2024-09-14T05:39:17.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 751052800 bytes 2024-09-14T05:39:17.677 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 803409920 bytes 2024-09-14T05:39:17.779 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 846225408 bytes 2024-09-14T05:39:17.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 883613696 bytes 2024-09-14T05:39:17.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 895214592 bytes 2024-09-14T05:39:18.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 909436928 bytes 2024-09-14T05:39:18.102 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 912795648 bytes 2024-09-14T05:39:18.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 942237696 bytes 2024-09-14T05:39:18.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 995185664 bytes 2024-09-14T05:39:18.402 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1053578240 bytes 2024-09-14T05:39:18.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1108944896 bytes 2024-09-14T05:39:18.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1189266432 bytes 2024-09-14T05:39:18.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1220147200 bytes 2024-09-14T05:39:18.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1263161344 bytes 2024-09-14T05:39:19.021 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1339109376 bytes 2024-09-14T05:39:19.097 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1344683008 bytes 2024-09-14T05:39:19.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1367817216 bytes 2024-09-14T05:39:19.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1406041088 bytes 2024-09-14T05:39:19.382 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1428578304 bytes 2024-09-14T05:39:19.500 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1485790208 bytes 2024-09-14T05:39:19.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1519265792 bytes 2024-09-14T05:39:19.711 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1573128192 bytes 2024-09-14T05:39:19.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1590234112 bytes 2024-09-14T05:39:19.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1595407360 bytes 2024-09-14T05:39:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1609220096 bytes 2024-09-14T05:39:20.054 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1651477504 bytes 2024-09-14T05:39:20.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1707134976 bytes 2024-09-14T05:39:20.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1731320832 bytes 2024-09-14T05:39:20.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1758254080 bytes 2024-09-14T05:39:20.456 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1806315520 bytes 2024-09-14T05:39:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1837301760 bytes 2024-09-14T05:39:20.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 1901849600 bytes 2024-09-14T05:39:20.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 1948087296 bytes 2024-09-14T05:39:20.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 1996553216 bytes 2024-09-14T05:39:21.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2033197056 bytes 2024-09-14T05:39:21.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2120567808 bytes 2024-09-14T05:39:21.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2133877760 bytes 2024-09-14T05:39:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2215621632 bytes 2024-09-14T05:39:21.457 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2250571776 bytes 2024-09-14T05:39:21.548 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2276820992 bytes 2024-09-14T05:39:21.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2336642048 bytes 2024-09-14T05:39:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2355322880 bytes 2024-09-14T05:39:21.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2405057536 bytes 2024-09-14T05:39:22.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2475310080 bytes 2024-09-14T05:39:22.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2533795840 bytes 2024-09-14T05:39:22.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2556521472 bytes 2024-09-14T05:39:22.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2557126656 bytes 2024-09-14T05:39:22.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2563147776 bytes 2024-09-14T05:39:22.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2622866432 bytes 2024-09-14T05:39:22.556 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2637532160 bytes 2024-09-14T05:39:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 2655123456 bytes 2024-09-14T05:39:22.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 2767025152 bytes 2024-09-14T05:39:22.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 2848062464 bytes 2024-09-14T05:39:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 2864644096 bytes 2024-09-14T05:39:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 2890861568 bytes 2024-09-14T05:39:23.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 2925056000 bytes 2024-09-14T05:39:23.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 2952455168 bytes 2024-09-14T05:39:23.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7300 IOs, 3048262656 bytes 2024-09-14T05:39:23.556 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7400 IOs, 3056960512 bytes 2024-09-14T05:39:23.682 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7500 IOs, 3116451840 bytes 2024-09-14T05:39:24.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7600 IOs, 3204149248 bytes 2024-09-14T05:39:24.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7700 IOs, 3223910400 bytes 2024-09-14T05:39:24.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7800 IOs, 3309260800 bytes 2024-09-14T05:39:24.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7900 IOs, 3367487488 bytes 2024-09-14T05:39:24.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8000 IOs, 3386924032 bytes 2024-09-14T05:39:24.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8100 IOs, 3480563712 bytes 2024-09-14T05:39:24.694 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8200 IOs, 3499239424 bytes 2024-09-14T05:39:24.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8300 IOs, 3534105600 bytes 2024-09-14T05:39:24.889 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8400 IOs, 3575002112 bytes 2024-09-14T05:39:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8500 IOs, 3620839424 bytes 2024-09-14T05:39:25.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8600 IOs, 3696128000 bytes 2024-09-14T05:39:25.247 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8700 IOs, 3773295616 bytes 2024-09-14T05:39:25.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8800 IOs, 3856478208 bytes 2024-09-14T05:39:25.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8900 IOs, 3940466688 bytes 2024-09-14T05:39:25.699 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9000 IOs, 4024005632 bytes 2024-09-14T05:39:25.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9100 IOs, 4104292352 bytes 2024-09-14T05:39:25.932 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9200 IOs, 4128781312 bytes 2024-09-14T05:39:26.062 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9300 IOs, 4198608896 bytes 2024-09-14T05:39:26.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9400 IOs, 4258176000 bytes 2024-09-14T05:39:26.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9500 IOs, 4298966016 bytes 2024-09-14T05:39:26.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9600 IOs, 4377992192 bytes 2024-09-14T05:39:26.500 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9700 IOs, 4413661184 bytes 2024-09-14T05:39:26.570 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:39:26.570 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:39:26.570 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:39:26.570 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:39:26.570 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-14T05:39:26.571 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:39:26.571 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:39:26.571 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:39:26.571 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-14T05:39:26.571 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local description_pattern= 2024-09-14T05:39:26.571 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local service_pattern= 2024-09-14T05:39:26.571 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:39:26.571 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:39:27.571 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-14T05:39:27.571 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:39:27.571 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:39:27.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:39:27.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-14T05:39:27.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local description_pattern= 2024-09-14T05:39:27.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local service_pattern= 2024-09-14T05:39:27.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status 2024-09-14T05:39:27.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CEPH_ARGS= 2024-09-14T05:39:27.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-14T05:39:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status='test: 2024-09-14T05:39:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: global_id: 77511074-da27-44b2-8b96-37f8b827a31e 2024-09-14T05:39:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: state: up+replaying 2024-09-14T05:39:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":526863360.0,"last_snapshot_bytes":535542784,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726292291,"remote_snapshot_timestamp":1726292291,"replay_state":"idle"} 2024-09-14T05:39:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: service: mirror.2 on smithi191 2024-09-14T05:39:27.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: last_update: 2024-09-14 05:39:05 2024-09-14T05:39:27.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: peer_sites: 2024-09-14T05:39:27.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: name: cluster2 2024-09-14T05:39:27.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: state: up+stopped 2024-09-14T05:39:27.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: description: local image is primary 2024-09-14T05:39:27.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: last_update: 2024-09-14 05:39:05' 2024-09-14T05:39:27.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^ state: .*up+replaying' 2024-09-14T05:39:27.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: state: up+replaying 2024-09-14T05:39:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^ description: .*' 2024-09-14T05:39:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":526863360.0,"last_snapshot_bytes":535542784,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726292291,"remote_snapshot_timestamp":1726292291,"replay_state":"idle"} 2024-09-14T05:39:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:39:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo up+replaying 2024-09-14T05:39:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^up+' 2024-09-14T05:39:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:up+replaying 2024-09-14T05:39:27.636 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep 'service: *mirror..* on ' 2024-09-14T05:39:27.637 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: service: mirror.2 on smithi191 2024-09-14T05:39:27.637 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local last_update 2024-09-14T05:39:27.637 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-14T05:39:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ last_update='2024-09-14 05:39:05' 2024-09-14T05:39:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-14 05:39:05' 2024-09-14T05:39:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:39:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:39:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:39:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-14T05:39:27.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local 'prev_last_update=2024-09-14 05:39:05' 2024-09-14T05:39:27.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status 2024-09-14T05:39:27.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CEPH_ARGS= 2024-09-14T05:39:27.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-14T05:39:27.703 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status='OKOKOK142164296mirror.1smithi19119.3.0-4775-g7f24e589falseOK42204289mirror.2smithi19119.3.0-4775-g7f24e589trueOK42244292mirror.3smithi19119.3.0-4775-g7f24e589falseOK43704372mirror.0smithi19119.3.0-4775-g7f24e589falseOKtest77511074-da27-44b2-8b96-37f8b827a31eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":526863360.0,"last_snapshot_bytes":535542784,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726292291,"remote_snapshot_timestamp":1726292291,"replay_state":"idle"}42204289mirror.2smithi1912024-09-14 05:39:05cluster29d38de88-cffd-4132-b7e8-809c69f2745aup+stoppedlocal image is primary2024-09-14 05:39:05' 2024-09-14T05:39:27.703 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local last_update state 2024-09-14T05:39:27.703 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-14T05:39:27.705 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ last_update='2024-09-14 05:39:05' 2024-09-14T05:39:27.705 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-14T05:39:27.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ state=up+replaying 2024-09-14T05:39:27.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo up+replaying 2024-09-14T05:39:27.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep up+replaying 2024-09-14T05:39:27.708 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:up+replaying 2024-09-14T05:39:27.709 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:39:27.709 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:39:27.709 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap6 2024-09-14T05:39:27.709 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror test snap6 2024-09-14T05:39:27.709 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:39:27.709 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:39:27.709 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:39:27.709 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap6 2024-09-14T05:39:27.709 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:39:42.446 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-14T05:39:42.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-14T05:39:42.447 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-09-14T05:39:42.448 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-14T05:39:42.446+0000 7f8cb752a640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 36750) UID: 1000 2024-09-14T05:39:50.448 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-14T05:39:50.448 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T05:39:50.448 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-14T05:39:50.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-14T05:39:50.449 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-14T05:39:50.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-14T05:39:50.452 INFO:tasks.rbd_mirror_thrash:waiting for 90 secs before thrashing 2024-09-14T05:39:55.446 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.2622328259460637) > weight (0.25) 2024-09-14T05:39:55.446 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.4794850522740567) > weight (0.25) 2024-09-14T05:39:55.446 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.7136449756042743) > weight (0.25) 2024-09-14T05:39:55.446 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.7291569623906892) > weight (0.25) 2024-09-14T05:39:55.446 INFO:tasks.rbd_mirror_thrash:waiting for 111 secs before thrashing 2024-09-14T05:39:58.038 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-14T05:39:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:39:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:39:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:39:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-14T05:39:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:39:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:39:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:39:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:39:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:39:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:39:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:39:59.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-14T05:39:59.040 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:39:59.040 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:39:59.040 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:39:59.040 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:39:59.040 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:39:59.040 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:39:59.041 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-14T05:39:59.041 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:39:59.041 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:39:59.041 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:39:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:39:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:39:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:39:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:39:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:39:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:39:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:39:59.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:39:59.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-14T05:39:59.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:39:59.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:39:59.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:39:59.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:39:59.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:39:59.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:39:59.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:39:59.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:39:59.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:39:59.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:39:59.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:39:59.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:39:59.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:39:59.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:39:59.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:39:59.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:39:59.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:39:59.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:39:59.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:39:59.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:39:59.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:39:59.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:39:59.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:39:59.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:39:59.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:39:59.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:39:59.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:39:59.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:39:59.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:39:59.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:39:59.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:39:59.067 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:39:59.067 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:39:59.067 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-14T05:39:59.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T05:39:59.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:39:59.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:39:59.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:39:59.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-14T05:39:59.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-14T05:39:59.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:39:59.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:39:59.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T05:39:59.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-14T05:39:59.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:39:59.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:39:59.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:39:59.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-14T05:39:59.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T05:39:59.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:39:59.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:39:59.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-14T05:39:59.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:39:59.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T05:39:59.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-14T05:39:59.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:39:59.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:39:59.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T05:39:59.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-14T05:39:59.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:39:59.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:39:59.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:39:59.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:39:59.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T05:39:59.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:39:59.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:39:59.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-14T05:39:59.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-14T05:39:59.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T05:39:59.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:39:59.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:39:59.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:39:59.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-14T05:39:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T05:39:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:39:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:39:59.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:39:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-14T05:39:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-14T05:39:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:39:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:39:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T05:39:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-14T05:39:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:39:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:39:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:39:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-14T05:39:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T05:39:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:39:59.245 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:39:59.245 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-14T05:39:59.245 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:39:59.245 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T05:39:59.245 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-14T05:39:59.245 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:39:59.245 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:39:59.245 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T05:39:59.245 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-14T05:39:59.245 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:39:59.245 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:39:59.245 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:39:59.245 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:39:59.245 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T05:39:59.246 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:39:59.246 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:39:59.246 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-14T05:39:59.246 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-14T05:39:59.246 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T05:39:59.262 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:39:59.262 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:39:59.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:39:59.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-14T05:39:59.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:39:59.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:39:59.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:39:59.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:39:59.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:39:59.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:39:59.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:39:59.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:39:59.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-14T05:39:59.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:39:59.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:39:59.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:39:59.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:39:59.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:39:59.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:39:59.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-14T05:39:59.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:39:59.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:39:59.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:39:59.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:39:59.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-test.status 2024-09-14T05:39:59.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-test.status 2024-09-14T05:39:59.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:39:59.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.status 2024-09-14T05:39:59.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-test.status 2024-09-14T05:39:59.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-test.status 2024-09-14T05:39:59.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:39:59.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:39:59.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-14T05:39:59.350 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:39:59.350 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:39:59.350 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:39:59.350 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-14T05:40:01.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 20 2024-09-14T05:40:01.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.status 2024-09-14T05:40:01.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:40:01.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:40:01.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:40:01.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.status 2024-09-14T05:40:01.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-14T05:40:01.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:40:01.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:40:01.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=20 2024-09-14T05:40:01.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:40:01.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:40:01.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:40:01.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:40:01.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:40:01.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:40:01.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:40:01.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:40:01.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T05:40:01.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:40:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:40:01.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:40:01.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:40:01.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:40:01.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:40:01.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:40:01.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 20 = '' 2024-09-14T05:40:01.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:40:01.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:40:01.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:40:01.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:40:01.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:40:01.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:40:01.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:40:01.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T05:40:01.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:40:01.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:40:01.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:40:01.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:40:01.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:40:01.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:40:01.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:40:01.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 20 = '' 2024-09-14T05:40:01.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:40:01.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:40:02.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:40:02.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:40:02.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:40:02.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:40:02.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:40:02.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T05:40:02.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:40:02.705 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:40:02.705 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:40:02.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:40:02.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:40:02.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:40:02.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:40:02.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 20 = '' 2024-09-14T05:40:02.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:40:02.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-14T05:40:04.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:40:04.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:40:04.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:40:04.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:40:04.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:40:04.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T05:40:04.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:40:04.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:40:04.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=20 2024-09-14T05:40:04.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 20 = 20 2024-09-14T05:40:04.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:40:04.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror test snap6 2024-09-14T05:40:04.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:40:04.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:40:04.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:40:04.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap6 2024-09-14T05:40:04.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:40:04.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:40:04.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:40:05.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/test@snap6 2024-09-14T05:40:05.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'test': 2024-09-14T05:40:05.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 512 MiB in 128 objects 2024-09-14T05:40:05.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:40:05.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 12 2024-09-14T05:40:05.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c1a41659ac 2024-09-14T05:40:05.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c1a41659ac 2024-09-14T05:40:05.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:40:05.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:40:05.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:40:05.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:40:05.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:35:08 2024 2024-09-14T05:40:05.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:38:16 2024 2024-09-14T05:40:05.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:35:08 2024 2024-09-14T05:40:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:40:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:40:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:40:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 77511074-da27-44b2-8b96-37f8b827a31e 2024-09-14T05:40:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:40:05.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:40:05.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snap5-clean ']' 2024-09-14T05:40:05.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap5-clean 2024-09-14T05:40:05.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:40:05.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:40:05.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap5-clean 2024-09-14T05:40:05.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:40:05.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:40:05.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:40:05.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:40:05.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:40:05.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:40:05.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:40:06.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:40:06.667 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:40:06.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:40:07.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:40:07.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap6 2024-09-14T05:40:07.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:40:07.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:40:07.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap6 2024-09-14T05:40:07.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:40:07.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:40:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:40:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:40:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:40:07.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:40:07.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:40:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:40:08.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:40:08.450 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:40:08.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:40:08.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ clean_snap_name=snap6-clean 2024-09-14T05:40:08.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror test snap6-clean 2024-09-14T05:40:08.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:40:08.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:40:08.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:40:08.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap6-clean 2024-09-14T05:40:08.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6-clean --debug-rbd=20 --debug-journaler=20 2024-09-14T05:40:08.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-14T05:40:08.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror test 2024-09-14T05:40:08.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:40:08.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:40:08.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:40:09.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:40:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=9 2024-09-14T05:40:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:40:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-14T05:40:09.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 26625024 bytes 2024-09-14T05:40:09.456 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 145929216 bytes 2024-09-14T05:40:09.561 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 189462528 bytes 2024-09-14T05:40:09.685 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 249006080 bytes 2024-09-14T05:40:09.781 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 276342784 bytes 2024-09-14T05:40:09.855 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 276954112 bytes 2024-09-14T05:40:09.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 316416000 bytes 2024-09-14T05:40:10.070 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 363866112 bytes 2024-09-14T05:40:10.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 431881216 bytes 2024-09-14T05:40:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 461310976 bytes 2024-09-14T05:40:10.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 500397056 bytes 2024-09-14T05:40:10.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 541032448 bytes 2024-09-14T05:40:10.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 554673152 bytes 2024-09-14T05:40:10.687 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 612937728 bytes 2024-09-14T05:40:10.798 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 661694464 bytes 2024-09-14T05:40:10.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 687373312 bytes 2024-09-14T05:40:11.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 741295104 bytes 2024-09-14T05:40:11.119 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 797950976 bytes 2024-09-14T05:40:11.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 840216576 bytes 2024-09-14T05:40:11.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 892328960 bytes 2024-09-14T05:40:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 922831872 bytes 2024-09-14T05:40:11.516 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 923430912 bytes 2024-09-14T05:40:11.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 962207744 bytes 2024-09-14T05:40:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1025029120 bytes 2024-09-14T05:40:11.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1077672960 bytes 2024-09-14T05:40:11.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1128078336 bytes 2024-09-14T05:40:12.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1144147968 bytes 2024-09-14T05:40:12.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1166944256 bytes 2024-09-14T05:40:12.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1338365952 bytes 2024-09-14T05:40:12.455 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1371454464 bytes 2024-09-14T05:40:12.562 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1404412928 bytes 2024-09-14T05:40:12.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1413496832 bytes 2024-09-14T05:40:12.731 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1456375808 bytes 2024-09-14T05:40:12.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1506221056 bytes 2024-09-14T05:40:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1569026048 bytes 2024-09-14T05:40:13.074 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1595516928 bytes 2024-09-14T05:40:13.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1634153472 bytes 2024-09-14T05:40:13.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1702910976 bytes 2024-09-14T05:40:13.396 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1716973568 bytes 2024-09-14T05:40:13.459 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1717748736 bytes 2024-09-14T05:40:13.621 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1819594752 bytes 2024-09-14T05:40:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1850831872 bytes 2024-09-14T05:40:13.848 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1935679488 bytes 2024-09-14T05:40:13.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1981497344 bytes 2024-09-14T05:40:14.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 2011191296 bytes 2024-09-14T05:40:14.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 2041821184 bytes 2024-09-14T05:40:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 2063906816 bytes 2024-09-14T05:40:14.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2067640320 bytes 2024-09-14T05:40:14.453 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2152883200 bytes 2024-09-14T05:40:14.571 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2208984064 bytes 2024-09-14T05:40:14.614 INFO:tasks.ceph.cluster2.osd.2.smithi191.stderr:2024-09-14T05:40:14.613+0000 7f3763311640 -1 reset not still connected to 0x55facc3ffba0 2024-09-14T05:40:14.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2229600256 bytes 2024-09-14T05:40:14.737 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2258847744 bytes 2024-09-14T05:40:14.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2266228736 bytes 2024-09-14T05:40:14.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2296951808 bytes 2024-09-14T05:40:14.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2330674176 bytes 2024-09-14T05:40:15.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2338110464 bytes 2024-09-14T05:40:15.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2373296128 bytes 2024-09-14T05:40:15.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2419183616 bytes 2024-09-14T05:40:15.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2433569792 bytes 2024-09-14T05:40:15.453 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2485691392 bytes 2024-09-14T05:40:15.588 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2576856064 bytes 2024-09-14T05:40:15.704 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2610681856 bytes 2024-09-14T05:40:15.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2651130880 bytes 2024-09-14T05:40:15.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2700535808 bytes 2024-09-14T05:40:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2755389440 bytes 2024-09-14T05:40:16.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 2764619776 bytes 2024-09-14T05:40:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 2815544320 bytes 2024-09-14T05:40:16.343 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 2862505984 bytes 2024-09-14T05:40:16.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 2944835584 bytes 2024-09-14T05:40:16.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 3005564928 bytes 2024-09-14T05:40:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 3040962560 bytes 2024-09-14T05:40:16.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 3065504768 bytes 2024-09-14T05:40:16.863 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7300 IOs, 3080455168 bytes 2024-09-14T05:40:17.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7400 IOs, 3174781952 bytes 2024-09-14T05:40:17.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7500 IOs, 3188957184 bytes 2024-09-14T05:40:17.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7600 IOs, 3197825024 bytes 2024-09-14T05:40:17.250 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7700 IOs, 3210955776 bytes 2024-09-14T05:40:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7800 IOs, 3289664512 bytes 2024-09-14T05:40:17.692 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7900 IOs, 3323807744 bytes 2024-09-14T05:40:17.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8000 IOs, 3358904320 bytes 2024-09-14T05:40:17.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8100 IOs, 3457721344 bytes 2024-09-14T05:40:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:40:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:40:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:40:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:40:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-14T05:40:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:40:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:40:18.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:40:18.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-14T05:40:18.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local description_pattern= 2024-09-14T05:40:18.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local service_pattern= 2024-09-14T05:40:18.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:40:18.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:40:19.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-14T05:40:19.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:40:19.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:40:19.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:40:19.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-14T05:40:19.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local description_pattern= 2024-09-14T05:40:19.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local service_pattern= 2024-09-14T05:40:19.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status 2024-09-14T05:40:19.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CEPH_ARGS= 2024-09-14T05:40:19.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-14T05:40:19.074 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status='test: 2024-09-14T05:40:19.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: global_id: 77511074-da27-44b2-8b96-37f8b827a31e 2024-09-14T05:40:19.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: state: up+replaying 2024-09-14T05:40:19.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: description: replaying, {"bytes_per_second":1398101.33,"bytes_per_snapshot":535669760.0,"last_snapshot_bytes":534514688,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726292400,"remote_snapshot_timestamp":1726292400,"replay_state":"idle"} 2024-09-14T05:40:19.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: service: mirror.2 on smithi191 2024-09-14T05:40:19.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: last_update: 2024-09-14 05:40:05 2024-09-14T05:40:19.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: peer_sites: 2024-09-14T05:40:19.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: name: cluster2 2024-09-14T05:40:19.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: state: up+stopped 2024-09-14T05:40:19.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: description: local image is primary 2024-09-14T05:40:19.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: last_update: 2024-09-14 05:40:05' 2024-09-14T05:40:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^ state: .*up+replaying' 2024-09-14T05:40:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: state: up+replaying 2024-09-14T05:40:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^ description: .*' 2024-09-14T05:40:19.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: description: replaying, {"bytes_per_second":1398101.33,"bytes_per_snapshot":535669760.0,"last_snapshot_bytes":534514688,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726292400,"remote_snapshot_timestamp":1726292400,"replay_state":"idle"} 2024-09-14T05:40:19.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:40:19.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo up+replaying 2024-09-14T05:40:19.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^up+' 2024-09-14T05:40:19.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:up+replaying 2024-09-14T05:40:19.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep 'service: *mirror..* on ' 2024-09-14T05:40:19.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: service: mirror.2 on smithi191 2024-09-14T05:40:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local last_update 2024-09-14T05:40:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-14T05:40:19.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ last_update='2024-09-14 05:40:05' 2024-09-14T05:40:19.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-14 05:40:05' 2024-09-14T05:40:19.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:40:19.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:40:19.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:40:19.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-14T05:40:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local 'prev_last_update=2024-09-14 05:40:05' 2024-09-14T05:40:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status 2024-09-14T05:40:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CEPH_ARGS= 2024-09-14T05:40:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-14T05:40:19.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status='OKOKOK142164296mirror.1smithi19119.3.0-4775-g7f24e589falseOK42204289mirror.2smithi19119.3.0-4775-g7f24e589trueOK42244292mirror.3smithi19119.3.0-4775-g7f24e589falseOK43704372mirror.0smithi19119.3.0-4775-g7f24e589falseOKtest77511074-da27-44b2-8b96-37f8b827a31eup+replayingreplaying, {"bytes_per_second":1398101.33,"bytes_per_snapshot":535669760.0,"last_snapshot_bytes":534514688,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726292400,"remote_snapshot_timestamp":1726292400,"replay_state":"idle"}42204289mirror.2smithi1912024-09-14 05:40:05cluster29d38de88-cffd-4132-b7e8-809c69f2745aup+stoppedlocal image is primary2024-09-14 05:40:05' 2024-09-14T05:40:19.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local last_update state 2024-09-14T05:40:19.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-14T05:40:19.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ last_update='2024-09-14 05:40:05' 2024-09-14T05:40:19.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-14T05:40:19.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ state=up+replaying 2024-09-14T05:40:19.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo up+replaying 2024-09-14T05:40:19.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep up+replaying 2024-09-14T05:40:19.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:up+replaying 2024-09-14T05:40:19.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:40:19.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:40:19.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap7 2024-09-14T05:40:19.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror test snap7 2024-09-14T05:40:19.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:40:19.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:40:19.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:40:19.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap7 2024-09-14T05:40:19.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:40:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-14T05:40:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:40:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:40:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:40:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-14T05:40:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:40:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:40:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:40:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:40:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:40:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:40:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:40:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-14T05:40:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:40:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:40:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:40:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:40:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:40:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:40:47.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-14T05:40:47.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:40:47.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:40:47.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:40:47.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:40:47.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:40:47.054 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:40:47.054 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:40:47.054 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:40:47.054 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:40:47.054 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:40:47.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:40:47.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-14T05:40:47.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:40:47.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:40:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:40:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:40:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:40:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:40:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:40:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:40:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:40:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:40:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:40:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:40:47.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:40:47.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:40:47.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:40:47.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:40:47.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:40:47.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:40:47.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:40:47.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:40:47.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:40:47.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:40:47.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:40:47.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:40:47.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:40:47.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:40:47.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:40:47.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:40:47.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:40:47.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:40:47.080 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:40:47.080 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:40:47.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:40:47.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-14T05:40:47.160 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T05:40:47.160 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:40:47.160 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:40:47.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:40:47.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-14T05:40:47.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-14T05:40:47.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:40:47.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:40:47.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T05:40:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-14T05:40:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:40:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:40:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:40:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-14T05:40:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T05:40:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:40:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:40:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-14T05:40:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:40:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T05:40:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-14T05:40:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:40:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:40:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T05:40:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-14T05:40:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:40:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:40:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:40:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:40:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T05:40:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:40:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:40:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-14T05:40:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-14T05:40:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T05:40:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:40:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:40:47.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:40:47.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-14T05:40:47.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T05:40:47.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:40:47.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:40:47.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:40:47.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-14T05:40:47.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-14T05:40:47.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:40:47.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:40:47.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T05:40:47.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-14T05:40:47.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:40:47.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:40:47.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:40:47.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-14T05:40:47.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T05:40:47.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:40:47.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:40:47.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-14T05:40:47.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:40:47.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T05:40:47.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-14T05:40:47.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:40:47.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:40:47.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T05:40:47.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-14T05:40:47.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:40:47.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:40:47.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:40:47.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:40:47.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T05:40:47.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:40:47.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:40:47.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-14T05:40:47.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-14T05:40:47.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T05:40:47.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:40:47.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:40:47.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:40:47.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-14T05:40:47.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:40:47.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:40:47.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:40:47.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:40:47.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:40:47.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:40:47.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:40:47.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:40:47.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-14T05:40:47.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:40:47.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:40:47.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:40:47.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:40:47.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:40:47.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:40:47.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-14T05:40:47.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:40:47.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:40:47.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:40:47.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:40:47.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-test.status 2024-09-14T05:40:47.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-test.status 2024-09-14T05:40:47.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:40:47.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.status 2024-09-14T05:40:47.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-test.status 2024-09-14T05:40:47.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-test.status 2024-09-14T05:40:47.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:40:47.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:40:47.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-14T05:40:47.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:40:47.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:40:47.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:40:47.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-14T05:40:49.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 23 2024-09-14T05:40:49.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.status 2024-09-14T05:40:49.080 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:40:49.080 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:40:49.080 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:40:49.080 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.status 2024-09-14T05:40:49.080 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-14T05:40:49.080 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:40:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:40:49.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=23 2024-09-14T05:40:49.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:40:49.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:40:49.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:40:49.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:40:49.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:40:49.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:40:49.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:40:49.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:40:49.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T05:40:49.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:40:49.408 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:40:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:40:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:40:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:40:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:40:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:40:49.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 23 = '' 2024-09-14T05:40:49.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:40:49.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:40:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:40:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:40:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:40:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:40:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:40:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T05:40:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:40:49.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:40:49.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:40:49.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:40:49.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:40:49.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:40:49.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:40:49.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 23 = '' 2024-09-14T05:40:49.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:40:49.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:40:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:40:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:40:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:40:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:40:50.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:40:50.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T05:40:50.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:40:50.719 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:40:50.719 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:40:50.720 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:40:50.720 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:40:50.720 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:40:50.721 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:40:50.721 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 23 = '' 2024-09-14T05:40:50.721 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:40:50.721 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-14T05:40:52.322 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:40:52.322 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:40:52.322 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:40:52.323 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:40:52.323 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:40:52.323 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T05:40:52.323 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:40:52.374 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:40:52.376 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=23 2024-09-14T05:40:52.377 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 23 = 23 2024-09-14T05:40:52.377 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:40:52.377 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror test snap7 2024-09-14T05:40:52.377 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:40:52.377 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:40:52.377 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:40:52.377 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap7 2024-09-14T05:40:52.377 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:40:52.377 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:40:52.377 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:40:53.377 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/test@snap7 2024-09-14T05:40:53.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'test': 2024-09-14T05:40:53.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 512 MiB in 128 objects 2024-09-14T05:40:53.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:40:53.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 14 2024-09-14T05:40:53.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c1a41659ac 2024-09-14T05:40:53.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c1a41659ac 2024-09-14T05:40:53.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:40:53.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:40:53.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:40:53.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:40:53.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:35:08 2024 2024-09-14T05:40:53.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:40:06 2024 2024-09-14T05:40:53.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:35:08 2024 2024-09-14T05:40:53.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:40:53.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:40:53.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:40:53.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 77511074-da27-44b2-8b96-37f8b827a31e 2024-09-14T05:40:53.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:40:53.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:40:53.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snap6-clean ']' 2024-09-14T05:40:53.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap6-clean 2024-09-14T05:40:53.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:40:53.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:40:53.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap6-clean 2024-09-14T05:40:53.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:40:53.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:40:53.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:40:53.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:40:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:40:53.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:40:53.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:40:54.636 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:40:54.646 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:40:54.857 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:40:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:40:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap7 2024-09-14T05:40:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:40:54.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:40:54.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap7 2024-09-14T05:40:54.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:40:54.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:40:54.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:40:54.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:40:54.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:40:55.500 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:40:55.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:40:56.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:40:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:40:56.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:40:56.550 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:40:56.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ clean_snap_name=snap7-clean 2024-09-14T05:40:56.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror test snap7-clean 2024-09-14T05:40:56.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:40:56.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:40:56.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:40:56.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap7-clean 2024-09-14T05:40:56.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7-clean --debug-rbd=20 --debug-journaler=20 2024-09-14T05:40:57.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-14T05:40:57.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror test 2024-09-14T05:40:57.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:40:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:40:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:40:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:40:57.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=10 2024-09-14T05:40:57.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:40:57.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-14T05:40:57.821 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 42811392 bytes 2024-09-14T05:40:57.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 63562752 bytes 2024-09-14T05:40:58.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 64197632 bytes 2024-09-14T05:40:58.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 91375616 bytes 2024-09-14T05:40:58.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 156313600 bytes 2024-09-14T05:40:58.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 203699200 bytes 2024-09-14T05:40:58.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 249942016 bytes 2024-09-14T05:40:58.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 259757056 bytes 2024-09-14T05:40:58.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 320165888 bytes 2024-09-14T05:40:58.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 343720960 bytes 2024-09-14T05:40:58.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 393480192 bytes 2024-09-14T05:40:58.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 405426176 bytes 2024-09-14T05:40:59.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 498577408 bytes 2024-09-14T05:40:59.287 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 578791424 bytes 2024-09-14T05:40:59.387 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 609341440 bytes 2024-09-14T05:40:59.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 626461696 bytes 2024-09-14T05:40:59.562 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 669677568 bytes 2024-09-14T05:40:59.646 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 670298112 bytes 2024-09-14T05:40:59.733 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 697039872 bytes 2024-09-14T05:40:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 697644032 bytes 2024-09-14T05:40:59.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 711952384 bytes 2024-09-14T05:40:59.969 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 740040704 bytes 2024-09-14T05:41:00.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 782710784 bytes 2024-09-14T05:41:00.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 789710848 bytes 2024-09-14T05:41:00.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 818854912 bytes 2024-09-14T05:41:00.407 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 914975744 bytes 2024-09-14T05:41:00.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 919529472 bytes 2024-09-14T05:41:00.619 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 976691200 bytes 2024-09-14T05:41:00.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1011756032 bytes 2024-09-14T05:41:00.856 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1101990912 bytes 2024-09-14T05:41:00.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1143569408 bytes 2024-09-14T05:41:01.091 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1215339520 bytes 2024-09-14T05:41:01.221 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1270556672 bytes 2024-09-14T05:41:01.323 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1298891776 bytes 2024-09-14T05:41:01.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1364600832 bytes 2024-09-14T05:41:01.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1471562752 bytes 2024-09-14T05:41:01.782 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1579446272 bytes 2024-09-14T05:41:01.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1612143616 bytes 2024-09-14T05:41:02.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1710230528 bytes 2024-09-14T05:41:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1760796672 bytes 2024-09-14T05:41:02.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1828824064 bytes 2024-09-14T05:41:02.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1842062336 bytes 2024-09-14T05:41:02.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1863584768 bytes 2024-09-14T05:41:02.545 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1935282176 bytes 2024-09-14T05:41:02.668 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1975886848 bytes 2024-09-14T05:41:02.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 2006671360 bytes 2024-09-14T05:41:02.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 2090008576 bytes 2024-09-14T05:41:02.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2110899200 bytes 2024-09-14T05:41:03.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2156201984 bytes 2024-09-14T05:41:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2181446656 bytes 2024-09-14T05:41:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2203578368 bytes 2024-09-14T05:41:03.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2229783552 bytes 2024-09-14T05:41:03.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2285487104 bytes 2024-09-14T05:41:03.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2314838016 bytes 2024-09-14T05:41:03.677 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2340049920 bytes 2024-09-14T05:41:03.796 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2404787200 bytes 2024-09-14T05:41:03.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2420242432 bytes 2024-09-14T05:41:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2484992000 bytes 2024-09-14T05:41:04.128 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2539228160 bytes 2024-09-14T05:41:04.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2580317184 bytes 2024-09-14T05:41:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2619464704 bytes 2024-09-14T05:41:04.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2621683712 bytes 2024-09-14T05:41:04.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2655187968 bytes 2024-09-14T05:41:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2674036736 bytes 2024-09-14T05:41:04.731 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2755065856 bytes 2024-09-14T05:41:04.816 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 2770292736 bytes 2024-09-14T05:41:04.891 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 2781518848 bytes 2024-09-14T05:41:05.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 2842629120 bytes 2024-09-14T05:41:05.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 2857667584 bytes 2024-09-14T05:41:05.225 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 2911376384 bytes 2024-09-14T05:41:05.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 2941490176 bytes 2024-09-14T05:41:05.403 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 2961968128 bytes 2024-09-14T05:41:05.499 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7300 IOs, 2991062016 bytes 2024-09-14T05:41:05.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7400 IOs, 3030464512 bytes 2024-09-14T05:41:05.787 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7500 IOs, 3148061696 bytes 2024-09-14T05:41:05.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7600 IOs, 3170972672 bytes 2024-09-14T05:41:05.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7700 IOs, 3196988416 bytes 2024-09-14T05:41:06.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7800 IOs, 3244036096 bytes 2024-09-14T05:41:06.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7900 IOs, 3352209408 bytes 2024-09-14T05:41:06.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8000 IOs, 3376348160 bytes 2024-09-14T05:41:06.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8100 IOs, 3399122944 bytes 2024-09-14T05:41:06.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8200 IOs, 3418390528 bytes 2024-09-14T05:41:06.638 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8300 IOs, 3457497088 bytes 2024-09-14T05:41:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8400 IOs, 3520308224 bytes 2024-09-14T05:41:06.852 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8500 IOs, 3543867392 bytes 2024-09-14T05:41:06.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8600 IOs, 3572752384 bytes 2024-09-14T05:41:07.018 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8700 IOs, 3573385216 bytes 2024-09-14T05:41:07.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8800 IOs, 3673919488 bytes 2024-09-14T05:41:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8900 IOs, 3714766848 bytes 2024-09-14T05:41:07.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9000 IOs, 3780251648 bytes 2024-09-14T05:41:07.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9100 IOs, 3793169408 bytes 2024-09-14T05:41:07.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:41:07.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:41:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:41:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:41:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-14T05:41:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:41:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:41:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:41:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-14T05:41:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local description_pattern= 2024-09-14T05:41:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local service_pattern= 2024-09-14T05:41:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:41:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:41:08.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-14T05:41:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:41:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:41:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:41:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-14T05:41:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local description_pattern= 2024-09-14T05:41:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local service_pattern= 2024-09-14T05:41:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status 2024-09-14T05:41:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CEPH_ARGS= 2024-09-14T05:41:08.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-14T05:41:08.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status='test: 2024-09-14T05:41:08.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: global_id: 77511074-da27-44b2-8b96-37f8b827a31e 2024-09-14T05:41:08.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: state: up+replaying 2024-09-14T05:41:08.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: description: replaying, {"bytes_per_second":17895594.67,"bytes_per_snapshot":535691264.0,"last_snapshot_bytes":536867840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726292448,"remote_snapshot_timestamp":1726292448,"replay_state":"idle"} 2024-09-14T05:41:08.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: service: mirror.2 on smithi191 2024-09-14T05:41:08.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: last_update: 2024-09-14 05:41:05 2024-09-14T05:41:08.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: peer_sites: 2024-09-14T05:41:08.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: name: cluster2 2024-09-14T05:41:08.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: state: up+stopped 2024-09-14T05:41:08.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: description: local image is primary 2024-09-14T05:41:08.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: last_update: 2024-09-14 05:41:05' 2024-09-14T05:41:08.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^ state: .*up+replaying' 2024-09-14T05:41:08.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: state: up+replaying 2024-09-14T05:41:08.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^ description: .*' 2024-09-14T05:41:08.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: description: replaying, {"bytes_per_second":17895594.67,"bytes_per_snapshot":535691264.0,"last_snapshot_bytes":536867840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726292448,"remote_snapshot_timestamp":1726292448,"replay_state":"idle"} 2024-09-14T05:41:08.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:41:08.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo up+replaying 2024-09-14T05:41:08.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^up+' 2024-09-14T05:41:08.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:up+replaying 2024-09-14T05:41:08.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep 'service: *mirror..* on ' 2024-09-14T05:41:08.637 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: service: mirror.2 on smithi191 2024-09-14T05:41:08.637 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local last_update 2024-09-14T05:41:08.637 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-14T05:41:08.639 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ last_update='2024-09-14 05:41:05' 2024-09-14T05:41:08.639 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-14 05:41:05' 2024-09-14T05:41:08.639 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:41:08.639 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:41:08.639 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:41:08.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-14T05:41:08.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local 'prev_last_update=2024-09-14 05:41:05' 2024-09-14T05:41:08.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status 2024-09-14T05:41:08.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CEPH_ARGS= 2024-09-14T05:41:08.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-14T05:41:08.700 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status='OKOKOK142164296mirror.1smithi19119.3.0-4775-g7f24e589falseOK42204289mirror.2smithi19119.3.0-4775-g7f24e589trueOK42244292mirror.3smithi19119.3.0-4775-g7f24e589falseOK43704372mirror.0smithi19119.3.0-4775-g7f24e589falseOKtest77511074-da27-44b2-8b96-37f8b827a31eup+replayingreplaying, {"bytes_per_second":17895594.67,"bytes_per_snapshot":535691264.0,"last_snapshot_bytes":536867840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726292448,"remote_snapshot_timestamp":1726292448,"replay_state":"idle"}42204289mirror.2smithi1912024-09-14 05:41:05cluster29d38de88-cffd-4132-b7e8-809c69f2745aup+stoppedlocal image is primary2024-09-14 05:41:05' 2024-09-14T05:41:08.700 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local last_update state 2024-09-14T05:41:08.700 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-14T05:41:08.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ last_update='2024-09-14 05:41:05' 2024-09-14T05:41:08.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-14T05:41:08.703 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ state=up+replaying 2024-09-14T05:41:08.703 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo up+replaying 2024-09-14T05:41:08.704 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep up+replaying 2024-09-14T05:41:08.704 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:up+replaying 2024-09-14T05:41:08.705 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:41:08.705 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:41:08.705 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap8 2024-09-14T05:41:08.705 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror test snap8 2024-09-14T05:41:08.705 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:41:08.705 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:41:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:41:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap8 2024-09-14T05:41:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:41:20.454 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8415390703851855) > weight (0.25) 2024-09-14T05:41:20.454 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5636518827234421) > weight (0.25) 2024-09-14T05:41:20.454 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.3997794265080872) > weight (0.25) 2024-09-14T05:41:20.454 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2024-09-14T05:41:20.455 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2024-09-14T05:41:20.455 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-09-14T05:41:20.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-14T05:41:20.455+0000 7efd6b79a640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 39979) UID: 1000 2024-09-14T05:41:25.456 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2024-09-14T05:41:25.456 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T05:41:25.457 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2024-09-14T05:41:25.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-09-14T05:41:25.457 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-09-14T05:41:25.460 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-09-14T05:41:25.460 INFO:tasks.rbd_mirror_thrash:waiting for 60 secs before thrashing 2024-09-14T05:41:39.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-14T05:41:39.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:41:39.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:41:39.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:41:39.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-14T05:41:39.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:41:39.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:41:39.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:41:39.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:41:39.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:41:39.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:41:39.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:41:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-14T05:41:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:41:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:41:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:41:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:41:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:41:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:41:40.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-14T05:41:40.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:41:40.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:41:40.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:41:40.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:41:40.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:41:40.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:41:40.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:41:40.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:41:40.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:41:40.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:41:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:41:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-14T05:41:40.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:41:40.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:41:40.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:41:40.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:41:40.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:41:40.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:41:40.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:41:40.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:41:40.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:41:40.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:41:40.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:41:40.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:41:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:41:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:41:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:41:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:41:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:41:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:41:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:41:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:41:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:41:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:41:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:41:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:41:40.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:41:40.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:41:40.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:41:40.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:41:40.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:41:40.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:41:40.069 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:41:40.069 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:41:40.070 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:41:40.070 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-14T05:41:40.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T05:41:40.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:41:40.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:41:40.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:41:40.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-14T05:41:40.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-14T05:41:40.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:41:40.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:41:40.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T05:41:40.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-14T05:41:40.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:41:40.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:41:40.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:41:40.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-14T05:41:40.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T05:41:40.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:41:40.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:41:40.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-14T05:41:40.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:41:40.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T05:41:40.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-14T05:41:40.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:41:40.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:41:40.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T05:41:40.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-14T05:41:40.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:41:40.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:41:40.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:41:40.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:41:40.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T05:41:40.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:41:40.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:41:40.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-14T05:41:40.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-14T05:41:40.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T05:41:40.171 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:41:40.171 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:41:40.172 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:41:40.172 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-14T05:41:40.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T05:41:40.245 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:41:40.245 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:41:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:41:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-14T05:41:40.247 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-14T05:41:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:41:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:41:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T05:41:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-14T05:41:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:41:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:41:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:41:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-14T05:41:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T05:41:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:41:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:41:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-14T05:41:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:41:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T05:41:40.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-14T05:41:40.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:41:40.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:41:40.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T05:41:40.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-14T05:41:40.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:41:40.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:41:40.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:41:40.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:41:40.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T05:41:40.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:41:40.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:41:40.250 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-14T05:41:40.250 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-14T05:41:40.250 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T05:41:40.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:41:40.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:41:40.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:41:40.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-14T05:41:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:41:40.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:41:40.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:41:40.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:41:40.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:41:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:41:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:41:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:41:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-14T05:41:40.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:41:40.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:41:40.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:41:40.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:41:40.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:41:40.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:41:40.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-14T05:41:40.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:41:40.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:41:40.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:41:40.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:41:40.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-test.status 2024-09-14T05:41:40.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-test.status 2024-09-14T05:41:40.352 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:41:40.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.status 2024-09-14T05:41:40.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-test.status 2024-09-14T05:41:40.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-test.status 2024-09-14T05:41:40.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:41:40.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:41:40.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-14T05:41:40.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:41:40.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:41:40.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:41:40.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-14T05:41:42.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 26 2024-09-14T05:41:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.status 2024-09-14T05:41:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:41:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:41:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:41:42.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.status 2024-09-14T05:41:42.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:41:42.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-14T05:41:42.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:41:42.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=26 2024-09-14T05:41:42.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:41:42.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:41:42.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:41:42.341 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:41:42.341 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:41:42.341 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:41:42.341 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:41:42.341 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:41:42.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T05:41:42.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:41:42.393 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:41:42.393 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:41:42.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:41:42.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:41:42.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:41:42.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:41:42.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 26 = '' 2024-09-14T05:41:42.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:41:42.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:41:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:41:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:41:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:41:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:41:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:41:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T05:41:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:41:42.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:41:42.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:41:42.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:41:42.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:41:42.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:41:42.855 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:41:42.855 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 26 = '' 2024-09-14T05:41:42.855 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:41:42.855 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:41:43.656 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:41:43.657 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:41:43.657 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:41:43.657 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:41:43.657 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:41:43.657 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T05:41:43.657 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:41:43.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:41:43.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:41:43.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:41:43.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:41:43.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:41:43.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:41:43.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 26 = '' 2024-09-14T05:41:43.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:41:43.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-14T05:41:45.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:41:45.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:41:45.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:41:45.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:41:45.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:41:45.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T05:41:45.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:41:45.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:41:45.370 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=26 2024-09-14T05:41:45.370 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 26 = 26 2024-09-14T05:41:45.370 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:41:45.370 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror test snap8 2024-09-14T05:41:45.370 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:41:45.370 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:41:45.370 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:41:45.370 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap8 2024-09-14T05:41:45.370 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:41:45.370 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:41:45.370 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:41:46.370 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/test@snap8 2024-09-14T05:41:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'test': 2024-09-14T05:41:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 512 MiB in 128 objects 2024-09-14T05:41:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:41:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 16 2024-09-14T05:41:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c1a41659ac 2024-09-14T05:41:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c1a41659ac 2024-09-14T05:41:46.424 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:41:46.424 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:41:46.424 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:41:46.424 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:41:46.424 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:35:08 2024 2024-09-14T05:41:46.424 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:40:06 2024 2024-09-14T05:41:46.424 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:35:08 2024 2024-09-14T05:41:46.424 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:41:46.424 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:41:46.424 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:41:46.424 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 77511074-da27-44b2-8b96-37f8b827a31e 2024-09-14T05:41:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:41:46.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:41:46.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snap7-clean ']' 2024-09-14T05:41:46.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap7-clean 2024-09-14T05:41:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:41:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:41:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap7-clean 2024-09-14T05:41:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:41:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:41:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:41:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:41:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:41:46.447 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.38048967724978056) > weight (0.25) 2024-09-14T05:41:46.447 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.5799448753500563) > weight (0.25) 2024-09-14T05:41:46.447 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.9312981293300895) > weight (0.25) 2024-09-14T05:41:46.447 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2024-09-14T05:41:46.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2024-09-14T05:41:46.448 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-09-14T05:41:46.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-14T05:41:46.448+0000 7fb134cb4640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 36732) UID: 1000 2024-09-14T05:41:46.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:41:46.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:41:47.448 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2024-09-14T05:41:47.449 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T05:41:47.449 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2024-09-14T05:41:47.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-09-14T05:41:47.449 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-09-14T05:41:47.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-09-14T05:41:47.452 INFO:tasks.rbd_mirror_thrash:waiting for 77 secs before thrashing 2024-09-14T05:41:47.624 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:41:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:41:47.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:41:47.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:41:47.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap8 2024-09-14T05:41:47.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:41:47.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:41:47.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap8 2024-09-14T05:41:47.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:41:47.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:41:47.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:41:47.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:41:47.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:41:48.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:41:48.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:41:49.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:41:49.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:41:49.363 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:41:49.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:41:49.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ clean_snap_name=snap8-clean 2024-09-14T05:41:49.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror test snap8-clean 2024-09-14T05:41:49.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:41:49.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:41:49.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:41:49.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap8-clean 2024-09-14T05:41:49.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8-clean --debug-rbd=20 --debug-journaler=20 2024-09-14T05:41:50.055 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-14T05:41:50.055 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror test 2024-09-14T05:41:50.055 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:41:50.055 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:41:50.056 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:41:50.056 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:41:50.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=5 2024-09-14T05:41:50.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:41:50.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-14T05:41:50.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 39536640 bytes 2024-09-14T05:41:50.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 125375488 bytes 2024-09-14T05:41:50.653 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 171326464 bytes 2024-09-14T05:41:50.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 199021568 bytes 2024-09-14T05:41:50.844 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 231159808 bytes 2024-09-14T05:41:50.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 280247296 bytes 2024-09-14T05:41:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 303587328 bytes 2024-09-14T05:41:51.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 350365696 bytes 2024-09-14T05:41:51.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 395254784 bytes 2024-09-14T05:41:51.387 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 439904256 bytes 2024-09-14T05:41:51.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 473193472 bytes 2024-09-14T05:41:51.611 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 527164416 bytes 2024-09-14T05:41:51.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 602529792 bytes 2024-09-14T05:41:51.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 645309440 bytes 2024-09-14T05:41:51.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 647454720 bytes 2024-09-14T05:41:52.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 666273792 bytes 2024-09-14T05:41:52.157 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 719363072 bytes 2024-09-14T05:41:52.271 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 755318784 bytes 2024-09-14T05:41:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 825900032 bytes 2024-09-14T05:41:52.515 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 861164544 bytes 2024-09-14T05:41:52.600 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 864585728 bytes 2024-09-14T05:41:52.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 916161536 bytes 2024-09-14T05:41:52.816 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 940620800 bytes 2024-09-14T05:41:52.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 941255680 bytes 2024-09-14T05:41:53.019 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1008152576 bytes 2024-09-14T05:41:53.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1057429504 bytes 2024-09-14T05:41:53.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1080602624 bytes 2024-09-14T05:41:53.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1101841408 bytes 2024-09-14T05:41:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1102462976 bytes 2024-09-14T05:41:53.552 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1169376256 bytes 2024-09-14T05:41:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1210217472 bytes 2024-09-14T05:41:53.758 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1245595648 bytes 2024-09-14T05:41:53.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1279190016 bytes 2024-09-14T05:41:53.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1295236096 bytes 2024-09-14T05:41:54.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1354270720 bytes 2024-09-14T05:41:54.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1362928640 bytes 2024-09-14T05:41:54.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1431595008 bytes 2024-09-14T05:41:54.380 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1464256512 bytes 2024-09-14T05:41:54.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1502019584 bytes 2024-09-14T05:41:54.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1565657088 bytes 2024-09-14T05:41:54.732 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1612813312 bytes 2024-09-14T05:41:54.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1656840192 bytes 2024-09-14T05:41:54.965 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1712690176 bytes 2024-09-14T05:41:55.062 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:41:55.062 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:41:55.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:41:55.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:41:55.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-14T05:41:55.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:41:55.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:41:55.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:41:55.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-14T05:41:55.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local description_pattern= 2024-09-14T05:41:55.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local service_pattern= 2024-09-14T05:41:55.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:41:55.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:41:56.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-14T05:41:56.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:41:56.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:41:56.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:41:56.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-14T05:41:56.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local description_pattern= 2024-09-14T05:41:56.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local service_pattern= 2024-09-14T05:41:56.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status 2024-09-14T05:41:56.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CEPH_ARGS= 2024-09-14T05:41:56.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-14T05:41:56.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status='test: 2024-09-14T05:41:56.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: global_id: 77511074-da27-44b2-8b96-37f8b827a31e 2024-09-14T05:41:56.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: state: up+replaying 2024-09-14T05:41:56.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":535691264.0,"last_snapshot_bytes":536867840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726292448,"remote_snapshot_timestamp":1726292448,"replay_state":"idle"} 2024-09-14T05:41:56.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: service: mirror.2 on smithi191 2024-09-14T05:41:56.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: last_update: 2024-09-14 05:41:35 2024-09-14T05:41:56.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: peer_sites: 2024-09-14T05:41:56.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: name: cluster2 2024-09-14T05:41:56.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: state: up+stopped 2024-09-14T05:41:56.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: description: local image is primary 2024-09-14T05:41:56.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: last_update: 2024-09-14 05:41:35' 2024-09-14T05:41:56.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^ state: .*up+replaying' 2024-09-14T05:41:56.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: state: up+replaying 2024-09-14T05:41:56.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^ description: .*' 2024-09-14T05:41:56.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":535691264.0,"last_snapshot_bytes":536867840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726292448,"remote_snapshot_timestamp":1726292448,"replay_state":"idle"} 2024-09-14T05:41:56.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:41:56.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo up+replaying 2024-09-14T05:41:56.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^up+' 2024-09-14T05:41:56.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:up+replaying 2024-09-14T05:41:56.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep 'service: *mirror..* on ' 2024-09-14T05:41:56.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: service: mirror.2 on smithi191 2024-09-14T05:41:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local last_update 2024-09-14T05:41:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-14T05:41:56.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ last_update='2024-09-14 05:41:35' 2024-09-14T05:41:56.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-14 05:41:35' 2024-09-14T05:41:56.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:41:56.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:41:56.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:41:56.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-14T05:41:56.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local 'prev_last_update=2024-09-14 05:41:35' 2024-09-14T05:41:56.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status 2024-09-14T05:41:56.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CEPH_ARGS= 2024-09-14T05:41:56.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-14T05:41:56.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status='OKOKOK142164296mirror.1smithi19119.3.0-4775-g7f24e589falseOK42204289mirror.2smithi19119.3.0-4775-g7f24e589trueOK43704372mirror.0smithi19119.3.0-4775-g7f24e589falseOK45564559mirror.3smithi19119.3.0-4775-g7f24e589falseOKtest77511074-da27-44b2-8b96-37f8b827a31eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":535691264.0,"last_snapshot_bytes":536867840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726292448,"remote_snapshot_timestamp":1726292448,"replay_state":"idle"}42204289mirror.2smithi1912024-09-14 05:41:35cluster29d38de88-cffd-4132-b7e8-809c69f2745aup+stoppedlocal image is primary2024-09-14 05:41:35' 2024-09-14T05:41:56.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local last_update state 2024-09-14T05:41:56.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-14T05:41:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ last_update='2024-09-14 05:41:35' 2024-09-14T05:41:56.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-14T05:41:56.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ state=up+replaying 2024-09-14T05:41:56.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo up+replaying 2024-09-14T05:41:56.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep up+replaying 2024-09-14T05:41:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:up+replaying 2024-09-14T05:41:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:41:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:41:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap9 2024-09-14T05:41:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror test snap9 2024-09-14T05:41:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:41:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:41:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:41:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap9 2024-09-14T05:41:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:42:25.461 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5842440318203648) > weight (0.25) 2024-09-14T05:42:25.462 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.9471612508317272) > weight (0.25) 2024-09-14T05:42:25.462 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.5028868034434169) > weight (0.25) 2024-09-14T05:42:25.462 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2024-09-14T05:42:25.463 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2024-09-14T05:42:25.464 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-09-14T05:42:25.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-14T05:42:25.464+0000 7f4aefda5640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 41857) UID: 1000 2024-09-14T05:42:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-14T05:42:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:42:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:42:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:42:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-14T05:42:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:42:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:42:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:42:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:42:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:42:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:42:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:42:27.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-14T05:42:27.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:42:27.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:42:27.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:42:27.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:42:27.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:42:27.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:42:27.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-14T05:42:27.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:42:27.062 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:42:27.062 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:42:27.062 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:42:27.062 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:42:27.062 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:42:27.062 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:42:27.062 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:42:27.062 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:42:27.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:42:27.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:42:27.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-14T05:42:27.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:42:27.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:42:27.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:42:27.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:42:27.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:42:27.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:42:27.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:42:27.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:42:27.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:42:27.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:42:27.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:42:27.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:42:27.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:42:27.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:42:27.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:42:27.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:42:27.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:42:27.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:42:27.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:42:27.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:42:27.067 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:42:27.067 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:42:27.067 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:42:27.067 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:42:27.067 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:42:27.067 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:42:27.067 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:42:27.067 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:42:27.067 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:42:27.068 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:42:27.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:42:27.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:42:27.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:42:27.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-14T05:42:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T05:42:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:42:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:42:27.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:42:27.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-14T05:42:27.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-14T05:42:27.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:42:27.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:42:27.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T05:42:27.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-14T05:42:27.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:42:27.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:42:27.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:42:27.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-14T05:42:27.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T05:42:27.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:42:27.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:42:27.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-14T05:42:27.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:42:27.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T05:42:27.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-14T05:42:27.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:42:27.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:42:27.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T05:42:27.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-14T05:42:27.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:42:27.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:42:27.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:42:27.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:42:27.171 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T05:42:27.171 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:42:27.171 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:42:27.171 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-14T05:42:27.171 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-14T05:42:27.171 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T05:42:27.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:42:27.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:42:27.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:42:27.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-14T05:42:27.262 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T05:42:27.262 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:42:27.262 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:42:27.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:42:27.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-14T05:42:27.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-14T05:42:27.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:42:27.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:42:27.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T05:42:27.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-14T05:42:27.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:42:27.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:42:27.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:42:27.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-14T05:42:27.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T05:42:27.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:42:27.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:42:27.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-14T05:42:27.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:42:27.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T05:42:27.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-14T05:42:27.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:42:27.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:42:27.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T05:42:27.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-14T05:42:27.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:42:27.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:42:27.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:42:27.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:42:27.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T05:42:27.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:42:27.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:42:27.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-14T05:42:27.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-14T05:42:27.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T05:42:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:42:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:42:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:42:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-14T05:42:27.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:42:27.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:42:27.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:42:27.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:42:27.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:42:27.368 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:42:27.368 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:42:27.368 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:42:27.368 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-14T05:42:27.369 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:42:27.369 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:42:27.369 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:42:27.369 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:42:27.369 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:42:27.369 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:42:27.369 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-14T05:42:27.369 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:42:27.369 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:42:27.369 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:42:27.369 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:42:27.369 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-test.status 2024-09-14T05:42:27.369 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-test.status 2024-09-14T05:42:27.370 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:42:27.371 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.status 2024-09-14T05:42:27.371 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-test.status 2024-09-14T05:42:27.371 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-test.status 2024-09-14T05:42:27.371 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:42:27.371 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:42:27.371 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-14T05:42:27.371 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:42:27.371 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:42:27.371 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:42:27.371 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-14T05:42:29.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 29 2024-09-14T05:42:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.status 2024-09-14T05:42:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:42:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:42:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:42:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.status 2024-09-14T05:42:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-14T05:42:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:42:29.158 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:42:29.160 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=29 2024-09-14T05:42:29.160 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:42:29.160 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:42:29.160 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:42:29.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:42:29.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:42:29.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:42:29.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:42:29.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:42:29.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T05:42:29.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:42:29.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:42:29.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:42:29.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:42:29.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:42:29.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:42:29.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:42:29.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 29 = '' 2024-09-14T05:42:29.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:42:29.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:42:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:42:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:42:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:42:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:42:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:42:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T05:42:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:42:29.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:42:29.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:42:29.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:42:29.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:42:29.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:42:29.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:42:29.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 29 = '' 2024-09-14T05:42:29.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:42:29.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:42:30.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:42:30.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:42:30.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:42:30.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:42:30.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:42:30.673 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T05:42:30.673 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:42:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:42:30.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:42:30.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:42:30.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:42:30.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:42:30.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:42:30.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 29 = '' 2024-09-14T05:42:30.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:42:30.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-14T05:42:31.464 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2024-09-14T05:42:31.464 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T05:42:31.465 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2024-09-14T05:42:31.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-09-14T05:42:31.465 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-09-14T05:42:31.467 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-09-14T05:42:31.468 INFO:tasks.rbd_mirror_thrash:waiting for 92 secs before thrashing 2024-09-14T05:42:32.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:42:32.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:42:32.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:42:32.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:42:32.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:42:32.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T05:42:32.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:42:32.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:42:32.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=29 2024-09-14T05:42:32.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 29 = 29 2024-09-14T05:42:32.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:42:32.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror test snap9 2024-09-14T05:42:32.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:42:32.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:42:32.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:42:32.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap9 2024-09-14T05:42:32.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:42:32.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:42:32.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:42:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/test@snap9 2024-09-14T05:42:33.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'test': 2024-09-14T05:42:33.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 512 MiB in 128 objects 2024-09-14T05:42:33.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:42:33.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 18 2024-09-14T05:42:33.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c1a41659ac 2024-09-14T05:42:33.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c1a41659ac 2024-09-14T05:42:33.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:42:33.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:42:33.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:42:33.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:42:33.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:35:08 2024 2024-09-14T05:42:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:41:47 2024 2024-09-14T05:42:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:35:08 2024 2024-09-14T05:42:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:42:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:42:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:42:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 77511074-da27-44b2-8b96-37f8b827a31e 2024-09-14T05:42:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:42:33.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:42:33.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snap8-clean ']' 2024-09-14T05:42:33.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap8-clean 2024-09-14T05:42:33.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:42:33.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:42:33.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap8-clean 2024-09-14T05:42:33.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:42:33.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:42:33.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:42:33.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:42:33.652 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:42:34.186 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:42:34.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:42:34.830 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:42:34.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:42:35.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:42:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:42:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap9 2024-09-14T05:42:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:42:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:42:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap9 2024-09-14T05:42:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:42:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:42:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:42:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:42:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:42:35.711 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:42:35.722 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:42:36.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:42:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:42:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:42:36.701 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:42:36.701 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ clean_snap_name=snap9-clean 2024-09-14T05:42:36.701 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror test snap9-clean 2024-09-14T05:42:36.701 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:42:36.701 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:42:36.701 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:42:36.701 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap9-clean 2024-09-14T05:42:36.701 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9-clean --debug-rbd=20 --debug-journaler=20 2024-09-14T05:42:37.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-14T05:42:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror test 2024-09-14T05:42:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:42:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:42:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:42:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:42:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=10 2024-09-14T05:42:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:42:37.589 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-14T05:42:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 57785344 bytes 2024-09-14T05:42:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 58598400 bytes 2024-09-14T05:42:38.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 131506176 bytes 2024-09-14T05:42:38.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 189753344 bytes 2024-09-14T05:42:38.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 236092416 bytes 2024-09-14T05:42:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 303624192 bytes 2024-09-14T05:42:38.654 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 326469632 bytes 2024-09-14T05:42:38.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 354289664 bytes 2024-09-14T05:42:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 428846080 bytes 2024-09-14T05:42:38.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 449940480 bytes 2024-09-14T05:42:39.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 527089664 bytes 2024-09-14T05:42:39.264 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 592932864 bytes 2024-09-14T05:42:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 643336192 bytes 2024-09-14T05:42:39.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 680724480 bytes 2024-09-14T05:42:39.600 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 728500224 bytes 2024-09-14T05:42:39.731 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 794693632 bytes 2024-09-14T05:42:39.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 826818560 bytes 2024-09-14T05:42:39.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 907910144 bytes 2024-09-14T05:42:40.068 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 923717632 bytes 2024-09-14T05:42:40.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 960142336 bytes 2024-09-14T05:42:40.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 1026216960 bytes 2024-09-14T05:42:40.401 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 1049571328 bytes 2024-09-14T05:42:40.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1050189824 bytes 2024-09-14T05:42:40.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1077946368 bytes 2024-09-14T05:42:40.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1157698560 bytes 2024-09-14T05:42:40.816 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1179228160 bytes 2024-09-14T05:42:40.929 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1226397696 bytes 2024-09-14T05:42:41.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1251420160 bytes 2024-09-14T05:42:41.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1348502528 bytes 2024-09-14T05:42:41.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1386638336 bytes 2024-09-14T05:42:41.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1464295424 bytes 2024-09-14T05:42:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1493810176 bytes 2024-09-14T05:42:41.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1542545408 bytes 2024-09-14T05:42:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1555625984 bytes 2024-09-14T05:42:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1618224128 bytes 2024-09-14T05:42:42.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1731097600 bytes 2024-09-14T05:42:42.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1800169472 bytes 2024-09-14T05:42:42.296 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1817390080 bytes 2024-09-14T05:42:42.399 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1856261120 bytes 2024-09-14T05:42:42.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1869244416 bytes 2024-09-14T05:42:42.578 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1893835776 bytes 2024-09-14T05:42:42.684 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1934209024 bytes 2024-09-14T05:42:42.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1953234944 bytes 2024-09-14T05:42:42.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1953831936 bytes 2024-09-14T05:42:42.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1984073728 bytes 2024-09-14T05:42:43.019 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1994957824 bytes 2024-09-14T05:42:43.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 1996793856 bytes 2024-09-14T05:42:43.220 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2031835136 bytes 2024-09-14T05:42:43.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2077500416 bytes 2024-09-14T05:42:43.440 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2113638400 bytes 2024-09-14T05:42:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2121133056 bytes 2024-09-14T05:42:43.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2174394368 bytes 2024-09-14T05:42:43.785 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2233696256 bytes 2024-09-14T05:42:43.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2261514240 bytes 2024-09-14T05:42:44.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2326288384 bytes 2024-09-14T05:42:44.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2373580800 bytes 2024-09-14T05:42:44.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2429779968 bytes 2024-09-14T05:42:44.368 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2467259392 bytes 2024-09-14T05:42:44.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2539905024 bytes 2024-09-14T05:42:44.588 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2541039616 bytes 2024-09-14T05:42:44.689 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2590465024 bytes 2024-09-14T05:42:44.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2613288960 bytes 2024-09-14T05:42:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2625816576 bytes 2024-09-14T05:42:45.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2654269440 bytes 2024-09-14T05:42:45.101 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2681548800 bytes 2024-09-14T05:42:45.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 2704795648 bytes 2024-09-14T05:42:45.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 2789775360 bytes 2024-09-14T05:42:45.458 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 2828288000 bytes 2024-09-14T05:42:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 2871143424 bytes 2024-09-14T05:42:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 2890602496 bytes 2024-09-14T05:42:45.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 2928045056 bytes 2024-09-14T05:42:45.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 2969760768 bytes 2024-09-14T05:42:46.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7300 IOs, 3002345472 bytes 2024-09-14T05:42:46.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7400 IOs, 3045699584 bytes 2024-09-14T05:42:46.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7500 IOs, 3061531648 bytes 2024-09-14T05:42:46.331 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7600 IOs, 3116993536 bytes 2024-09-14T05:42:46.435 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7700 IOs, 3148513280 bytes 2024-09-14T05:42:46.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7800 IOs, 3235111936 bytes 2024-09-14T05:42:46.728 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7900 IOs, 3306086400 bytes 2024-09-14T05:42:46.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8000 IOs, 3371503616 bytes 2024-09-14T05:42:46.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8100 IOs, 3377120256 bytes 2024-09-14T05:42:47.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8200 IOs, 3392095232 bytes 2024-09-14T05:42:47.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8300 IOs, 3419768832 bytes 2024-09-14T05:42:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8400 IOs, 3526459392 bytes 2024-09-14T05:42:47.422 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8500 IOs, 3566140416 bytes 2024-09-14T05:42:47.550 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8600 IOs, 3621723136 bytes 2024-09-14T05:42:47.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:42:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:42:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:42:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:42:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-14T05:42:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:42:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:42:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:42:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-14T05:42:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local description_pattern= 2024-09-14T05:42:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local service_pattern= 2024-09-14T05:42:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:42:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:42:48.599 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-14T05:42:48.599 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:42:48.599 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:42:48.599 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:42:48.599 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-14T05:42:48.599 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local description_pattern= 2024-09-14T05:42:48.599 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local service_pattern= 2024-09-14T05:42:48.599 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status 2024-09-14T05:42:48.600 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CEPH_ARGS= 2024-09-14T05:42:48.600 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-14T05:42:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status='test: 2024-09-14T05:42:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: global_id: 77511074-da27-44b2-8b96-37f8b827a31e 2024-09-14T05:42:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: state: up+replaying 2024-09-14T05:42:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: description: replaying, {"bytes_per_second":1398101.33,"bytes_per_snapshot":529812992.0,"last_snapshot_bytes":524343296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726292548,"remote_snapshot_timestamp":1726292548,"replay_state":"idle"} 2024-09-14T05:42:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: service: mirror.2 on smithi191 2024-09-14T05:42:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: last_update: 2024-09-14 05:42:35 2024-09-14T05:42:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: peer_sites: 2024-09-14T05:42:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: name: cluster2 2024-09-14T05:42:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: state: up+stopped 2024-09-14T05:42:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: description: local image is primary 2024-09-14T05:42:48.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: last_update: 2024-09-14 05:42:35' 2024-09-14T05:42:48.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^ state: .*up+replaying' 2024-09-14T05:42:48.668 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: state: up+replaying 2024-09-14T05:42:48.668 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^ description: .*' 2024-09-14T05:42:48.669 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: description: replaying, {"bytes_per_second":1398101.33,"bytes_per_snapshot":529812992.0,"last_snapshot_bytes":524343296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726292548,"remote_snapshot_timestamp":1726292548,"replay_state":"idle"} 2024-09-14T05:42:48.669 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:42:48.669 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo up+replaying 2024-09-14T05:42:48.669 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep '^up+' 2024-09-14T05:42:48.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:up+replaying 2024-09-14T05:42:48.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep 'service: *mirror..* on ' 2024-09-14T05:42:48.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: service: mirror.2 on smithi191 2024-09-14T05:42:48.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local last_update 2024-09-14T05:42:48.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-14T05:42:48.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ last_update='2024-09-14 05:42:35' 2024-09-14T05:42:48.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-14 05:42:35' 2024-09-14T05:42:48.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:42:48.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:42:48.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:42:48.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state_pattern=up+replaying 2024-09-14T05:42:48.673 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local 'prev_last_update=2024-09-14 05:42:35' 2024-09-14T05:42:48.673 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status 2024-09-14T05:42:48.673 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ CEPH_ARGS= 2024-09-14T05:42:48.673 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-14T05:42:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status='OKOKOK142164296mirror.1smithi19119.3.0-4775-g7f24e589falseOK42204289mirror.2smithi19119.3.0-4775-g7f24e589trueOK43704372mirror.0smithi19119.3.0-4775-g7f24e589falseOK45564559mirror.3smithi19119.3.0-4775-g7f24e589falseOKtest77511074-da27-44b2-8b96-37f8b827a31eup+replayingreplaying, {"bytes_per_second":1398101.33,"bytes_per_snapshot":529812992.0,"last_snapshot_bytes":524343296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726292548,"remote_snapshot_timestamp":1726292548,"replay_state":"idle"}42204289mirror.2smithi1912024-09-14 05:42:35cluster29d38de88-cffd-4132-b7e8-809c69f2745aup+stoppedlocal image is primary2024-09-14 05:42:35' 2024-09-14T05:42:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local last_update state 2024-09-14T05:42:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-14T05:42:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ last_update='2024-09-14 05:42:35' 2024-09-14T05:42:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-14T05:42:48.739 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ state=up+replaying 2024-09-14T05:42:48.739 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo up+replaying 2024-09-14T05:42:48.739 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep up+replaying 2024-09-14T05:42:48.740 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:up+replaying 2024-09-14T05:42:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:42:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:42:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap10 2024-09-14T05:42:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror test snap10 2024-09-14T05:42:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:42:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:42:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:42:48.742 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap10 2024-09-14T05:42:48.742 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:43:04.453 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-14T05:43:04.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-14T05:43:04.454 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-09-14T05:43:04.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-14T05:43:04.454+0000 7f5c5aa6c640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 39286) UID: 1000 2024-09-14T05:43:10.455 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-14T05:43:10.455 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T05:43:10.456 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-14T05:43:10.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-14T05:43:10.456 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-14T05:43:10.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-14T05:43:10.459 INFO:tasks.rbd_mirror_thrash:waiting for 72 secs before thrashing 2024-09-14T05:43:18.624 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-14T05:43:18.624 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:43:18.624 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:43:18.625 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:43:18.625 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-14T05:43:18.625 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:43:18.625 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:43:18.625 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:43:18.625 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:43:18.625 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:43:18.625 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:43:18.625 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:43:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-14T05:43:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:43:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:43:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:43:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:43:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:43:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:43:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-14T05:43:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:43:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:43:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:43:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:43:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:43:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:43:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:43:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:43:19.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:43:19.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:43:19.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:43:19.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-14T05:43:19.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:43:19.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:43:19.631 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:43:19.631 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:43:19.631 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:43:19.631 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:43:19.631 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:43:19.631 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:43:19.631 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:43:19.631 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:43:19.631 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:43:19.631 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:43:19.631 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:43:19.631 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:43:19.631 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:43:19.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:43:19.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:43:19.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:43:19.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:43:19.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:43:19.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:43:19.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:43:19.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:43:19.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:43:19.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:43:19.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:43:19.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:43:19.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:43:19.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:43:19.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:43:19.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:43:19.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:43:19.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:43:19.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-14T05:43:19.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T05:43:19.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:43:19.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:43:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:43:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-14T05:43:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-14T05:43:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:43:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:43:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T05:43:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-14T05:43:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:43:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:43:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:43:19.730 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-14T05:43:19.730 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T05:43:19.730 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:43:19.730 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:43:19.730 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-14T05:43:19.730 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:43:19.730 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T05:43:19.730 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-14T05:43:19.730 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:43:19.730 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:43:19.730 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T05:43:19.730 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-14T05:43:19.730 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:43:19.731 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:43:19.731 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:43:19.731 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:43:19.731 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T05:43:19.731 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:43:19.731 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:43:19.731 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-14T05:43:19.731 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-14T05:43:19.731 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T05:43:19.747 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:43:19.748 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:43:19.748 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:43:19.748 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-14T05:43:19.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T05:43:19.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:43:19.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:43:19.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:43:19.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-14T05:43:19.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-14T05:43:19.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:43:19.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:43:19.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T05:43:19.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-14T05:43:19.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:43:19.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:43:19.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:43:19.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-14T05:43:19.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T05:43:19.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:43:19.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:43:19.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-14T05:43:19.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:43:19.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T05:43:19.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-14T05:43:19.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:43:19.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:43:19.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T05:43:19.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-14T05:43:19.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:43:19.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:43:19.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:43:19.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:43:19.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T05:43:19.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:43:19.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:43:19.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-14T05:43:19.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-14T05:43:19.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T05:43:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:43:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:43:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:43:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-14T05:43:19.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:43:19.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:43:19.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:43:19.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:43:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:43:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:43:19.925 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:43:19.925 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:43:19.925 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-14T05:43:19.925 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:43:19.925 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:43:19.925 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:43:19.925 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:43:19.925 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:43:19.925 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:43:19.925 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-14T05:43:19.925 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:43:19.926 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:43:19.926 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:43:19.926 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:43:19.926 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-test.status 2024-09-14T05:43:19.926 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-test.status 2024-09-14T05:43:19.926 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:43:19.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.status 2024-09-14T05:43:19.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-test.status 2024-09-14T05:43:19.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-test.status 2024-09-14T05:43:19.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:43:19.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:43:19.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-14T05:43:19.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:43:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:43:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:43:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-14T05:43:21.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 32 2024-09-14T05:43:21.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.status 2024-09-14T05:43:21.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:43:21.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:43:21.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:43:21.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.status 2024-09-14T05:43:21.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-14T05:43:21.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:43:21.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:43:21.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=32 2024-09-14T05:43:21.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:43:21.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:43:21.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:43:21.931 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:43:21.931 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:43:21.931 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:43:21.931 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:43:21.931 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:43:21.931 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T05:43:21.931 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:43:21.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:43:21.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:43:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:43:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:43:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:43:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:43:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 32 = '' 2024-09-14T05:43:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:43:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:43:22.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:43:22.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:43:22.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:43:22.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:43:22.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:43:22.392 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T05:43:22.392 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:43:22.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:43:22.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:43:22.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:43:22.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:43:22.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:43:22.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:43:22.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 32 = '' 2024-09-14T05:43:22.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:43:22.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:43:23.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:43:23.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:43:23.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:43:23.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:43:23.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:43:23.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T05:43:23.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:43:23.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:43:23.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:43:23.343 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:43:23.343 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:43:23.343 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:43:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:43:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 32 = '' 2024-09-14T05:43:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:43:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-14T05:43:24.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:43:24.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:43:24.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:43:24.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:43:24.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:43:24.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T05:43:24.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:43:25.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:43:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=32 2024-09-14T05:43:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 32 = 32 2024-09-14T05:43:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:43:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror test snap10 2024-09-14T05:43:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:43:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:43:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:43:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap10 2024-09-14T05:43:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:43:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:43:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:43:26.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/test@snap10 2024-09-14T05:43:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'test': 2024-09-14T05:43:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 512 MiB in 128 objects 2024-09-14T05:43:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:43:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 20 2024-09-14T05:43:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c1a41659ac 2024-09-14T05:43:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c1a41659ac 2024-09-14T05:43:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:43:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:43:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:43:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:43:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:35:08 2024 2024-09-14T05:43:26.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:41:47 2024 2024-09-14T05:43:26.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:35:08 2024 2024-09-14T05:43:26.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:43:26.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:43:26.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:43:26.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 77511074-da27-44b2-8b96-37f8b827a31e 2024-09-14T05:43:26.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:43:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:43:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snap9-clean ']' 2024-09-14T05:43:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap9-clean 2024-09-14T05:43:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:43:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:43:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap9-clean 2024-09-14T05:43:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:43:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:43:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:26.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:43:26.613 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:43:26.625 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:27.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:43:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:27.599 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:27.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:43:27.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap10 2024-09-14T05:43:27.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:43:27.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:43:27.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap10 2024-09-14T05:43:27.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:43:27.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:43:27.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:27.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:27.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:43:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:43:28.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:28.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:43:28.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:29.097 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:43:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ clean_snap_name=snap10-clean 2024-09-14T05:43:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror test snap10-clean 2024-09-14T05:43:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:43:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:43:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:43:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap10-clean 2024-09-14T05:43:29.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10-clean --debug-rbd=20 --debug-journaler=20 2024-09-14T05:43:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-14T05:43:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:43:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:43:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:43:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-14T05:43:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:43:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:43:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:43:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:43:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:43:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:43:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:43:31.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-14T05:43:31.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:43:31.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:43:31.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:43:31.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:43:31.035 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:43:31.035 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:43:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-14T05:43:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:43:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:43:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:43:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:43:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:43:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:43:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:43:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:43:31.037 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:43:31.037 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:43:31.038 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:43:31.038 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-14T05:43:31.038 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:43:31.038 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:43:31.038 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:43:31.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:43:31.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:43:31.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:43:31.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:43:31.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:43:31.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:43:31.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:43:31.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:43:31.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:43:31.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:43:31.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:43:31.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:43:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:43:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:43:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:43:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:43:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:43:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:43:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:43:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:43:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:43:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:43:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:43:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:43:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:43:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:43:31.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:43:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:43:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:43:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:43:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-14T05:43:31.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T05:43:31.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:43:31.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:43:31.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:43:31.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-14T05:43:31.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-14T05:43:31.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:43:31.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:43:31.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T05:43:31.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-14T05:43:31.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:43:31.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:43:31.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:43:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-14T05:43:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T05:43:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:43:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:43:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-14T05:43:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:43:31.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T05:43:31.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-14T05:43:31.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:43:31.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:43:31.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T05:43:31.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-14T05:43:31.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:43:31.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:43:31.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:43:31.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:43:31.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T05:43:31.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:43:31.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:43:31.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-14T05:43:31.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-14T05:43:31.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T05:43:31.161 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:43:31.161 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:43:31.161 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:43:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-14T05:43:31.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-14T05:43:31.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:43:31.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:43:31.237 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:43:31.237 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-14T05:43:31.237 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-14T05:43:31.237 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:43:31.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:43:31.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T05:43:31.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-14T05:43:31.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:43:31.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:43:31.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:43:31.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-14T05:43:31.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T05:43:31.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:43:31.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:43:31.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-14T05:43:31.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:43:31.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T05:43:31.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-14T05:43:31.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:43:31.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:43:31.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T05:43:31.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-14T05:43:31.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:43:31.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:43:31.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:43:31.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:43:31.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T05:43:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:43:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:43:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-14T05:43:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-14T05:43:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T05:43:31.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:43:31.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:43:31.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:43:31.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-14T05:43:31.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:43:31.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:43:31.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:43:31.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:43:31.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:43:31.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:43:31.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:43:31.343 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:43:31.343 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-14T05:43:31.343 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:43:31.343 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:43:31.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:43:31.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:43:31.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:43:31.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:43:31.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-14T05:43:31.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:43:31.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:43:31.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:43:31.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:43:31.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-test.status 2024-09-14T05:43:31.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-test.status 2024-09-14T05:43:31.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:43:31.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.status 2024-09-14T05:43:31.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-test.status 2024-09-14T05:43:31.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-test.status 2024-09-14T05:43:31.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:43:31.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:43:31.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-14T05:43:31.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:43:31.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:43:31.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:43:31.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-14T05:43:33.021 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 34 2024-09-14T05:43:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.status 2024-09-14T05:43:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:43:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:43:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:43:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.status 2024-09-14T05:43:33.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-14T05:43:33.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:43:33.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:43:33.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=34 2024-09-14T05:43:33.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:43:33.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:43:33.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:43:33.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:43:33.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:43:33.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:43:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:43:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:43:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T05:43:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:43:33.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:43:33.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:43:33.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:43:33.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:43:33.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:43:33.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:43:33.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 34 = '' 2024-09-14T05:43:33.392 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:43:33.392 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:43:33.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:43:33.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:43:33.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:43:33.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:43:33.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:43:33.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T05:43:33.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:43:33.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:43:33.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:43:33.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:43:33.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:43:33.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:43:33.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:43:33.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 34 = '' 2024-09-14T05:43:33.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:43:33.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:43:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:43:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:43:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:43:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:43:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.status 2024-09-14T05:43:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-14T05:43:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:43:34.703 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:43:34.705 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=34 2024-09-14T05:43:34.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 34 = 34 2024-09-14T05:43:34.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:43:34.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror test snap10-clean 2024-09-14T05:43:34.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:43:34.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:43:34.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:43:34.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap10-clean 2024-09-14T05:43:34.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:43:34.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:43:34.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:43:35.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/test@snap10-clean 2024-09-14T05:43:35.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'test': 2024-09-14T05:43:35.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 512 MiB in 128 objects 2024-09-14T05:43:35.761 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:43:35.761 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 21 2024-09-14T05:43:35.761 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c1a41659ac 2024-09-14T05:43:35.761 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c1a41659ac 2024-09-14T05:43:35.761 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:43:35.761 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:43:35.762 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:43:35.762 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:43:35.762 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:35:08 2024 2024-09-14T05:43:35.762 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:43:26 2024 2024-09-14T05:43:35.762 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:35:08 2024 2024-09-14T05:43:35.762 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:43:35.762 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:43:35.762 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:43:35.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 77511074-da27-44b2-8b96-37f8b827a31e 2024-09-14T05:43:35.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:43:35.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:43:35.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ seq 1 10 2024-09-14T05:43:35.765 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-14T05:43:35.765 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap1 2024-09-14T05:43:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap1 2024-09-14T05:43:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:43:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:43:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap1 2024-09-14T05:43:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:43:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:43:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:43:36.330 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:43:36.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:43:37.096 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:37.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:43:37.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap1-clean 2024-09-14T05:43:37.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap1-clean 2024-09-14T05:43:37.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:43:37.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:43:37.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap1-clean 2024-09-14T05:43:37.421 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:43:37.421 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:43:37.421 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:37.421 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:37.421 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:43:37.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:43:37.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:38.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:43:38.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:38.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:43:38.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-14T05:43:38.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap2 2024-09-14T05:43:38.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap2 2024-09-14T05:43:38.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:43:38.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:43:38.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap2 2024-09-14T05:43:38.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:43:38.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:43:38.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:38.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:38.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:43:39.498 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:43:39.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:40.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:43:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:40.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:40.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:43:40.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap2-clean 2024-09-14T05:43:40.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap2-clean 2024-09-14T05:43:40.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:43:40.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:43:40.610 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap2-clean 2024-09-14T05:43:40.610 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:43:40.610 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:43:40.610 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:40.610 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:40.610 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:43:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:43:41.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:42.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:43:42.074 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:42.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:42.399 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:43:42.400 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-14T05:43:42.400 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap3 2024-09-14T05:43:42.400 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap3 2024-09-14T05:43:42.400 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:43:42.400 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:43:42.400 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap3 2024-09-14T05:43:42.400 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:43:42.400 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:43:42.400 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:42.400 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:42.401 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:43:42.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:43:42.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:43.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:43:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:43.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:43.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:43:43.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap3-clean 2024-09-14T05:43:43.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap3-clean 2024-09-14T05:43:43.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:43:43.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:43:43.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap3-clean 2024-09-14T05:43:43.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:43:43.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:43:43.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:43.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:43.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:43:44.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:43:44.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:45.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:43:45.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:45.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:43:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-14T05:43:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap4 2024-09-14T05:43:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap4 2024-09-14T05:43:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:43:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:43:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap4 2024-09-14T05:43:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:43:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:43:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:43:46.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:43:46.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:43:46.696 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:46.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:47.022 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:43:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap4-clean 2024-09-14T05:43:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap4-clean 2024-09-14T05:43:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:43:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:43:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap4-clean 2024-09-14T05:43:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:43:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:43:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:47.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:43:47.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:43:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:43:48.330 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:48.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:48.660 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:43:48.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-14T05:43:48.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap5 2024-09-14T05:43:48.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap5 2024-09-14T05:43:48.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:43:48.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:43:48.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap5 2024-09-14T05:43:48.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:43:48.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:43:48.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:48.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:48.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:43:49.398 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:43:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:50.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:43:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:50.330 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:50.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:43:50.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap5-clean 2024-09-14T05:43:50.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap5-clean 2024-09-14T05:43:50.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:43:50.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:43:50.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap5-clean 2024-09-14T05:43:50.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:43:50.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:43:50.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:50.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:50.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:43:50.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:43:51.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:51.724 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:43:51.730 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:52.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:43:52.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-14T05:43:52.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap6 2024-09-14T05:43:52.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap6 2024-09-14T05:43:52.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:43:52.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:43:52.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap6 2024-09-14T05:43:52.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:43:52.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:43:52.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:52.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:52.062 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:43:52.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:43:52.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:43:53.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:53.564 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:53.681 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:43:53.681 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap6-clean 2024-09-14T05:43:53.681 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap6-clean 2024-09-14T05:43:53.681 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:43:53.681 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:43:53.681 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap6-clean 2024-09-14T05:43:53.681 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:43:53.681 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:43:53.682 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:53.682 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:53.682 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:43:54.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:43:54.226 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:54.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:43:54.951 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:55.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:43:55.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-14T05:43:55.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap7 2024-09-14T05:43:55.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap7 2024-09-14T05:43:55.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:43:55.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:43:55.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap7 2024-09-14T05:43:55.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:43:55.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:43:55.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:55.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:55.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:43:55.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:43:55.837 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:56.584 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:43:56.596 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:56.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:56.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:43:56.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap7-clean 2024-09-14T05:43:56.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap7-clean 2024-09-14T05:43:56.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:43:56.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:43:56.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap7-clean 2024-09-14T05:43:56.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:43:56.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:43:56.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:56.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:56.929 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:43:57.458 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:43:57.469 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:58.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:43:58.185 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:58.396 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:58.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:43:58.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-14T05:43:58.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap8 2024-09-14T05:43:58.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap8 2024-09-14T05:43:58.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:43:58.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:43:58.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap8 2024-09-14T05:43:58.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:43:58.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:43:58.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:58.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:58.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:43:59.037 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:43:59.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:59.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:43:59.765 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:43:59.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:44:00.095 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:44:00.095 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap8-clean 2024-09-14T05:44:00.095 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap8-clean 2024-09-14T05:44:00.096 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:44:00.096 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:44:00.096 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap8-clean 2024-09-14T05:44:00.096 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:44:00.096 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:44:00.096 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:44:00.096 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:44:00.096 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:44:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:44:00.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:44:01.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:44:01.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:44:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:44:01.676 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:44:01.676 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-14T05:44:01.677 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap9 2024-09-14T05:44:01.677 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap9 2024-09-14T05:44:01.677 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:44:01.677 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:44:01.677 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap9 2024-09-14T05:44:01.677 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:44:01.677 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:44:01.677 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:44:01.677 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:44:01.677 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:44:02.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:44:02.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:44:02.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:44:02.947 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:44:03.160 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:44:03.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:44:03.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap9-clean 2024-09-14T05:44:03.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap9-clean 2024-09-14T05:44:03.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:44:03.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:44:03.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap9-clean 2024-09-14T05:44:03.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:44:03.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:44:03.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:44:03.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:44:03.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:44:03.469 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.2551725827910333) > weight (0.25) 2024-09-14T05:44:03.469 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5383185444262466) > weight (0.25) 2024-09-14T05:44:03.469 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-09-14T05:44:03.469 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-09-14T05:44:03.470 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-09-14T05:44:03.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-14T05:44:03.469+0000 7fd5d8abf640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 36780) UID: 1000 2024-09-14T05:44:03.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:44:03.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:44:04.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:44:04.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:44:04.797 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:44:04.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:44:04.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-14T05:44:04.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap10 2024-09-14T05:44:04.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap10 2024-09-14T05:44:04.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:44:04.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:44:04.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap10 2024-09-14T05:44:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:44:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:44:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:44:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:44:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:44:05.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:44:05.458 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:44:06.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:44:06.157 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:44:06.371 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:44:06.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:44:06.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap10-clean 2024-09-14T05:44:06.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror test snap10-clean 2024-09-14T05:44:06.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:44:06.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:44:06.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap10-clean 2024-09-14T05:44:06.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:44:06.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:44:06.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:44:06.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:44:06.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10-clean /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export 2024-09-14T05:44:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:44:07.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10-clean /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:44:07.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-14T05:44:07.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:44:07.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-test.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-test.export 2024-09-14T05:44:08.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:44:08.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ seq 1 10 2024-09-14T05:44:08.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-14T05:44:08.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap1 2024-09-14T05:44:08.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror test snap1 2024-09-14T05:44:08.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:44:08.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:44:08.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:44:08.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap1 2024-09-14T05:44:08.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1 2024-09-14T05:44:08.470 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-09-14T05:44:08.470 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T05:44:08.471 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-09-14T05:44:08.471 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-14T05:44:08.471 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-14T05:44:08.474 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-14T05:44:08.474 INFO:tasks.rbd_mirror_thrash:waiting for 60 secs before thrashing 2024-09-14T05:44:09.062 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T05:44:09.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap1-clean 2024-09-14T05:44:09.091 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror test snap1-clean 2024-09-14T05:44:09.091 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:44:09.091 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:44:09.091 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:44:09.091 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap1-clean 2024-09-14T05:44:09.091 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1-clean 2024-09-14T05:44:10.073 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T05:44:10.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-14T05:44:10.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap2 2024-09-14T05:44:10.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror test snap2 2024-09-14T05:44:10.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:44:10.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:44:10.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:44:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap2 2024-09-14T05:44:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2 2024-09-14T05:44:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T05:44:11.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap2-clean 2024-09-14T05:44:11.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror test snap2-clean 2024-09-14T05:44:11.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:44:11.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:44:11.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:44:11.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap2-clean 2024-09-14T05:44:11.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2-clean 2024-09-14T05:44:12.080 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T05:44:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-14T05:44:12.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap3 2024-09-14T05:44:12.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror test snap3 2024-09-14T05:44:12.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:44:12.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:44:12.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:44:12.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap3 2024-09-14T05:44:12.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3 2024-09-14T05:44:13.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T05:44:13.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap3-clean 2024-09-14T05:44:13.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror test snap3-clean 2024-09-14T05:44:13.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:44:13.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:44:13.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:44:13.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap3-clean 2024-09-14T05:44:13.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3-clean 2024-09-14T05:44:14.095 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T05:44:14.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-14T05:44:14.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap4 2024-09-14T05:44:14.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror test snap4 2024-09-14T05:44:14.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:44:14.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:44:14.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:44:14.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap4 2024-09-14T05:44:14.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4 2024-09-14T05:44:15.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T05:44:15.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap4-clean 2024-09-14T05:44:15.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror test snap4-clean 2024-09-14T05:44:15.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:44:15.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:44:15.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:44:15.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap4-clean 2024-09-14T05:44:15.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4-clean 2024-09-14T05:44:16.101 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T05:44:16.128 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-14T05:44:16.128 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap5 2024-09-14T05:44:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror test snap5 2024-09-14T05:44:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:44:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:44:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:44:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap5 2024-09-14T05:44:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5 2024-09-14T05:44:17.097 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T05:44:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap5-clean 2024-09-14T05:44:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror test snap5-clean 2024-09-14T05:44:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:44:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:44:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:44:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap5-clean 2024-09-14T05:44:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5-clean 2024-09-14T05:44:18.012 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T05:44:18.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-14T05:44:18.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap6 2024-09-14T05:44:18.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror test snap6 2024-09-14T05:44:18.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:44:18.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:44:18.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:44:18.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap6 2024-09-14T05:44:18.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6 2024-09-14T05:44:19.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T05:44:19.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap6-clean 2024-09-14T05:44:19.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror test snap6-clean 2024-09-14T05:44:19.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:44:19.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:44:19.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:44:19.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap6-clean 2024-09-14T05:44:19.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6-clean 2024-09-14T05:44:20.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T05:44:20.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-14T05:44:20.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap7 2024-09-14T05:44:20.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror test snap7 2024-09-14T05:44:20.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:44:20.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:44:20.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:44:20.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap7 2024-09-14T05:44:20.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7 2024-09-14T05:44:21.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T05:44:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap7-clean 2024-09-14T05:44:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror test snap7-clean 2024-09-14T05:44:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:44:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:44:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:44:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap7-clean 2024-09-14T05:44:21.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7-clean 2024-09-14T05:44:22.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T05:44:22.158 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-14T05:44:22.158 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap8 2024-09-14T05:44:22.158 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror test snap8 2024-09-14T05:44:22.158 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:44:22.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:44:22.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:44:22.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap8 2024-09-14T05:44:22.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8 2024-09-14T05:44:22.460 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6541256198622268) > weight (0.25) 2024-09-14T05:44:22.460 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.4349975521846152) > weight (0.25) 2024-09-14T05:44:22.461 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.6590159142119805) > weight (0.25) 2024-09-14T05:44:22.461 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.5921665665484319) > weight (0.25) 2024-09-14T05:44:22.461 INFO:tasks.rbd_mirror_thrash:waiting for 95 secs before thrashing 2024-09-14T05:44:23.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T05:44:23.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap8-clean 2024-09-14T05:44:23.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror test snap8-clean 2024-09-14T05:44:23.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:44:23.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:44:23.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:44:23.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap8-clean 2024-09-14T05:44:23.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8-clean 2024-09-14T05:44:24.128 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T05:44:24.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-14T05:44:24.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap9 2024-09-14T05:44:24.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror test snap9 2024-09-14T05:44:24.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:44:24.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:44:24.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:44:24.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap9 2024-09-14T05:44:24.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9 2024-09-14T05:44:25.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T05:44:25.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap9-clean 2024-09-14T05:44:25.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror test snap9-clean 2024-09-14T05:44:25.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:44:25.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:44:25.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:44:25.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap9-clean 2024-09-14T05:44:25.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9-clean 2024-09-14T05:44:26.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T05:44:26.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-14T05:44:26.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap10 2024-09-14T05:44:26.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror test snap10 2024-09-14T05:44:26.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:44:26.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:44:26.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:44:26.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap10 2024-09-14T05:44:26.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10 2024-09-14T05:44:27.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T05:44:27.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap10-clean 2024-09-14T05:44:27.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror test snap10-clean 2024-09-14T05:44:27.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:44:27.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:44:27.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:44:27.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap10-clean 2024-09-14T05:44:27.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10-clean 2024-09-14T05:44:28.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T05:44:28.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror test 2024-09-14T05:44:28.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:44:28.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:44:28.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:44:28.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T05:44:28.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T05:44:28.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror test 2024-09-14T05:44:28.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:44:28.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:44:28.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:44:28.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/test 2024-09-14T05:44:28.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/test 2024-09-14T05:44:28.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T05:44:28.275+0000 7fd089f8c640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ee84edd860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T05:44:28.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T05:44:28.275+0000 7fd090799640 -1 rbd::mirror::ImageReplayer: 0x55ee84eee500 [3/77511074-da27-44b2-8b96-37f8b827a31e] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T05:44:30.331 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-14T05:44:30.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:44:30.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror test deleted 2024-09-14T05:44:30.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:44:30.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:44:30.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:44:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T05:44:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T05:44:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:44:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T05:44:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror test 2024-09-14T05:44:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:44:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T05:44:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=test 2024-09-14T05:44:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-14T05:44:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T05:44:30.370 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image test: (2) No such file or directory 2024-09-14T05:44:30.373 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T05:44:30.373 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T05:44:30.373 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T05:44:30.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror test deleted '' 2024-09-14T05:44:30.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:44:30.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:44:30.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=test 2024-09-14T05:44:30.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T05:44:30.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T05:44:30.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T05:44:30.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T05:44:30.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror test 2024-09-14T05:44:30.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:44:30.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T05:44:30.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=test 2024-09-14T05:44:30.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-14T05:44:30.477 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T05:44:30.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image test: (2) No such file or directory 2024-09-14T05:44:30.514 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T05:44:30.514 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T05:44:30.514 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T05:44:30.514 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:44:30.514 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ testlog 'TEST: create many images' 2024-09-14T05:44:30.514 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ tee -a /tmp/tmp.sXArEbIrwv/rbd-mirror.test.log 2024-09-14T05:44:30.514 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ date '+%F %T' 2024-09-14T05:44:30.515 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo 2024-09-14 05:44:30 TEST: create many images 2024-09-14T05:44:30.515 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:2024-09-14 05:44:30 TEST: create many images 2024-09-14T05:44:30.515 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ snap_name=snap 2024-09-14T05:44:30.515 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ seq 1 50 2024-09-14T05:44:30.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:44:30.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_1 2024-09-14T05:44:30.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_1 snapshot 128M 2024-09-14T05:44:30.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:44:30.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:44:30.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:44:30.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:44:30.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-14T05:44:30.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:44:30.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:44:30.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-14T05:44:30.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:44:30.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_1 128M 2024-09-14T05:44:30.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:44:30.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:44:30.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:44:30.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:44:30.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-14T05:44:30.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:44:30.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T05:44:30.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-14T05:44:30.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-14T05:44:30.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:44:30.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_1 2024-09-14T05:44:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T05:44:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:44:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-14T05:44:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:44:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_1 snapshot 2024-09-14T05:44:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:44:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:44:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-14T05:44:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:44:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2024-09-14T05:44:31.158 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T05:44:31.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2024-09-14T05:44:31.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_1': 2024-09-14T05:44:31.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:44:31.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:44:31.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T05:44:31.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 1268f879e8fe 2024-09-14T05:44:31.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.1268f879e8fe 2024-09-14T05:44:31.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:44:31.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T05:44:31.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:44:31.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:44:31.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:44:30 2024 2024-09-14T05:44:31.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:44:30 2024 2024-09-14T05:44:31.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:44:30 2024 2024-09-14T05:44:31.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:44:31.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:44:31.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: cc05da34-933d-4ab7-a8cd-8e584cdd7250 2024-09-14T05:44:31.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T05:44:31.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:44:31.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_1 2024-09-14T05:44:31.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:44:31.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:44:31.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-14T05:44:31.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:44:31.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=13 2024-09-14T05:44:31.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:44:31.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:44:31.454 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 115427328 bytes 2024-09-14T05:44:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 151217152 bytes 2024-09-14T05:44:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 184914944 bytes 2024-09-14T05:44:31.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 185519104 bytes 2024-09-14T05:44:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 209156096 bytes 2024-09-14T05:44:31.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 216146944 bytes 2024-09-14T05:44:32.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 265979904 bytes 2024-09-14T05:44:32.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 327649280 bytes 2024-09-14T05:44:32.228 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 355438592 bytes 2024-09-14T05:44:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 388077568 bytes 2024-09-14T05:44:32.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 446911488 bytes 2024-09-14T05:44:32.550 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 477789184 bytes 2024-09-14T05:44:32.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 521412608 bytes 2024-09-14T05:44:32.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 566788096 bytes 2024-09-14T05:44:32.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 641863680 bytes 2024-09-14T05:44:33.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 667416576 bytes 2024-09-14T05:44:33.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 746491904 bytes 2024-09-14T05:44:33.270 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 800461824 bytes 2024-09-14T05:44:33.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 815708160 bytes 2024-09-14T05:44:33.440 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 843194368 bytes 2024-09-14T05:44:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 903797760 bytes 2024-09-14T05:44:33.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 935731200 bytes 2024-09-14T05:44:33.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1024173056 bytes 2024-09-14T05:44:33.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1055495168 bytes 2024-09-14T05:44:34.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1112411136 bytes 2024-09-14T05:44:34.119 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1129492480 bytes 2024-09-14T05:44:34.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1151851520 bytes 2024-09-14T05:44:34.329 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1203569664 bytes 2024-09-14T05:44:34.458 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1267534848 bytes 2024-09-14T05:44:34.578 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1320441856 bytes 2024-09-14T05:44:34.657 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1340896256 bytes 2024-09-14T05:44:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1354859520 bytes 2024-09-14T05:44:34.899 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1477493760 bytes 2024-09-14T05:44:34.991 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1494378496 bytes 2024-09-14T05:44:35.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1514914816 bytes 2024-09-14T05:44:35.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1546869760 bytes 2024-09-14T05:44:35.301 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1607281664 bytes 2024-09-14T05:44:35.402 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1640067072 bytes 2024-09-14T05:44:35.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1648990208 bytes 2024-09-14T05:44:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1681431552 bytes 2024-09-14T05:44:35.731 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1752386560 bytes 2024-09-14T05:44:35.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1772699648 bytes 2024-09-14T05:44:35.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1822500864 bytes 2024-09-14T05:44:36.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1855556608 bytes 2024-09-14T05:44:36.221 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1967015936 bytes 2024-09-14T05:44:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 2032556032 bytes 2024-09-14T05:44:36.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 2092053504 bytes 2024-09-14T05:44:36.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2157347840 bytes 2024-09-14T05:44:36.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2218544128 bytes 2024-09-14T05:44:36.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2277906432 bytes 2024-09-14T05:44:36.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2300876800 bytes 2024-09-14T05:44:37.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2352724992 bytes 2024-09-14T05:44:37.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2410333184 bytes 2024-09-14T05:44:37.327 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2472060928 bytes 2024-09-14T05:44:37.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2492772352 bytes 2024-09-14T05:44:37.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2493373440 bytes 2024-09-14T05:44:37.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2516183040 bytes 2024-09-14T05:44:37.703 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2570621952 bytes 2024-09-14T05:44:37.770 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2578796544 bytes 2024-09-14T05:44:37.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2625408000 bytes 2024-09-14T05:44:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2681542656 bytes 2024-09-14T05:44:38.333 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2761990144 bytes 2024-09-14T05:44:38.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2799684608 bytes 2024-09-14T05:44:38.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2835863552 bytes 2024-09-14T05:44:38.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2907976704 bytes 2024-09-14T05:44:38.778 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 2942925824 bytes 2024-09-14T05:44:38.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 2978856960 bytes 2024-09-14T05:44:38.952 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 2989246464 bytes 2024-09-14T05:44:39.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 2989846528 bytes 2024-09-14T05:44:39.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 3036681216 bytes 2024-09-14T05:44:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 3077122048 bytes 2024-09-14T05:44:39.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 3143875584 bytes 2024-09-14T05:44:39.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7300 IOs, 3176509440 bytes 2024-09-14T05:44:39.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7400 IOs, 3208269824 bytes 2024-09-14T05:44:39.745 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7500 IOs, 3312792576 bytes 2024-09-14T05:44:39.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7600 IOs, 3345179648 bytes 2024-09-14T05:44:39.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7700 IOs, 3351431168 bytes 2024-09-14T05:44:40.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7800 IOs, 3397376000 bytes 2024-09-14T05:44:40.185 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7900 IOs, 3474291712 bytes 2024-09-14T05:44:40.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8000 IOs, 3541879808 bytes 2024-09-14T05:44:40.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8100 IOs, 3585501184 bytes 2024-09-14T05:44:40.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8200 IOs, 3606805504 bytes 2024-09-14T05:44:40.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8300 IOs, 3662580736 bytes 2024-09-14T05:44:40.733 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8400 IOs, 3681255424 bytes 2024-09-14T05:44:40.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8500 IOs, 3729948672 bytes 2024-09-14T05:44:40.969 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8600 IOs, 3776714752 bytes 2024-09-14T05:44:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8700 IOs, 3795185664 bytes 2024-09-14T05:44:41.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8800 IOs, 3817125888 bytes 2024-09-14T05:44:41.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8900 IOs, 3863574528 bytes 2024-09-14T05:44:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9000 IOs, 3896316928 bytes 2024-09-14T05:44:41.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9100 IOs, 3947764736 bytes 2024-09-14T05:44:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9200 IOs, 4012722176 bytes 2024-09-14T05:44:41.728 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9300 IOs, 4060187648 bytes 2024-09-14T05:44:41.799 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9400 IOs, 4060826624 bytes 2024-09-14T05:44:41.899 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9500 IOs, 4098491392 bytes 2024-09-14T05:44:41.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9600 IOs, 4099102720 bytes 2024-09-14T05:44:42.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9700 IOs, 4150666240 bytes 2024-09-14T05:44:42.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9800 IOs, 4173970432 bytes 2024-09-14T05:44:42.287 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9900 IOs, 4201062400 bytes 2024-09-14T05:44:42.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10000 IOs, 4278599680 bytes 2024-09-14T05:44:42.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10100 IOs, 4321768448 bytes 2024-09-14T05:44:42.667 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10200 IOs, 4377834496 bytes 2024-09-14T05:44:42.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10300 IOs, 4437715968 bytes 2024-09-14T05:44:42.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10400 IOs, 4461460480 bytes 2024-09-14T05:44:43.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10500 IOs, 4539492352 bytes 2024-09-14T05:44:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10600 IOs, 4582837248 bytes 2024-09-14T05:44:43.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10700 IOs, 4661927936 bytes 2024-09-14T05:44:43.423 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10800 IOs, 4705004544 bytes 2024-09-14T05:44:43.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10900 IOs, 4738161664 bytes 2024-09-14T05:44:43.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:11000 IOs, 4750162944 bytes 2024-09-14T05:44:43.697 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:11100 IOs, 4773135360 bytes 2024-09-14T05:44:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:11200 IOs, 4796456960 bytes 2024-09-14T05:44:43.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:11300 IOs, 4820652032 bytes 2024-09-14T05:44:43.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:11400 IOs, 4843896832 bytes 2024-09-14T05:44:44.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:11500 IOs, 4877441024 bytes 2024-09-14T05:44:44.185 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:11600 IOs, 4918192128 bytes 2024-09-14T05:44:44.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:44:44.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:44:44.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:44:44.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:44:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:44:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_2 2024-09-14T05:44:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_2 snapshot 128M 2024-09-14T05:44:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:44:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:44:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:44:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:44:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-14T05:44:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:44:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:44:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-14T05:44:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:44:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_2 128M 2024-09-14T05:44:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:44:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:44:44.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:44:44.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:44:44.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-14T05:44:44.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:44:44.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T05:44:44.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-14T05:44:44.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-14T05:44:44.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:44:44.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_2 2024-09-14T05:44:44.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T05:44:44.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:44:44.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-14T05:44:44.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:44:44.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_2 snapshot 2024-09-14T05:44:44.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:44:44.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:44:44.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-14T05:44:44.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:44:44.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2024-09-14T05:44:45.040 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T05:44:45.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2024-09-14T05:44:45.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_2': 2024-09-14T05:44:45.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:44:45.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:44:45.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T05:44:45.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 1272d20e95e9 2024-09-14T05:44:45.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.1272d20e95e9 2024-09-14T05:44:45.088 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:44:45.088 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T05:44:45.088 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:44:45.088 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:44:45.088 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:44:44 2024 2024-09-14T05:44:45.088 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:44:44 2024 2024-09-14T05:44:45.088 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:44:44 2024 2024-09-14T05:44:45.088 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:44:45.088 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:44:45.088 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: a89e8edb-c81f-4b3c-b444-29c2fea9908b 2024-09-14T05:44:45.088 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T05:44:45.091 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:44:45.091 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_2 2024-09-14T05:44:45.091 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:44:45.091 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:44:45.091 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-14T05:44:45.091 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:44:45.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=8 2024-09-14T05:44:45.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:44:45.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:44:45.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 50393088 bytes 2024-09-14T05:44:45.368 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 85723136 bytes 2024-09-14T05:44:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 125094912 bytes 2024-09-14T05:44:45.596 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 173501440 bytes 2024-09-14T05:44:45.673 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 180106240 bytes 2024-09-14T05:44:45.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 201972736 bytes 2024-09-14T05:44:45.852 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 220069888 bytes 2024-09-14T05:44:45.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 264855552 bytes 2024-09-14T05:44:46.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 292472832 bytes 2024-09-14T05:44:46.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 314597376 bytes 2024-09-14T05:44:46.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 343051264 bytes 2024-09-14T05:44:46.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 388621312 bytes 2024-09-14T05:44:46.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 418331648 bytes 2024-09-14T05:44:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 418962432 bytes 2024-09-14T05:44:46.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 442835968 bytes 2024-09-14T05:44:46.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 485774336 bytes 2024-09-14T05:44:46.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 555378688 bytes 2024-09-14T05:44:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 628769792 bytes 2024-09-14T05:44:47.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 659847168 bytes 2024-09-14T05:44:47.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 706865152 bytes 2024-09-14T05:44:47.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 707492864 bytes 2024-09-14T05:44:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 733483008 bytes 2024-09-14T05:44:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 841785344 bytes 2024-09-14T05:44:47.668 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 894546944 bytes 2024-09-14T05:44:47.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 940814336 bytes 2024-09-14T05:44:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 972701696 bytes 2024-09-14T05:44:48.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1044204544 bytes 2024-09-14T05:44:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1077712896 bytes 2024-09-14T05:44:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1141125120 bytes 2024-09-14T05:44:48.471 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1307153408 bytes 2024-09-14T05:44:48.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1307760640 bytes 2024-09-14T05:44:48.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1351875584 bytes 2024-09-14T05:44:48.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1417197568 bytes 2024-09-14T05:44:48.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1446460416 bytes 2024-09-14T05:44:48.947 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1470949376 bytes 2024-09-14T05:44:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1492045824 bytes 2024-09-14T05:44:49.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1540158464 bytes 2024-09-14T05:44:49.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1587654656 bytes 2024-09-14T05:44:49.370 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1624936448 bytes 2024-09-14T05:44:49.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1688453120 bytes 2024-09-14T05:44:49.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1722218496 bytes 2024-09-14T05:44:49.698 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1752739840 bytes 2024-09-14T05:44:49.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1802875904 bytes 2024-09-14T05:44:49.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1836011520 bytes 2024-09-14T05:44:50.011 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1882421248 bytes 2024-09-14T05:44:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1933003776 bytes 2024-09-14T05:44:50.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 1987804160 bytes 2024-09-14T05:44:50.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2010208256 bytes 2024-09-14T05:44:50.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2105109504 bytes 2024-09-14T05:44:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2135518208 bytes 2024-09-14T05:44:50.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2162752512 bytes 2024-09-14T05:44:50.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2219503616 bytes 2024-09-14T05:44:50.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2310275072 bytes 2024-09-14T05:44:51.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2335336448 bytes 2024-09-14T05:44:51.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2338887680 bytes 2024-09-14T05:44:51.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2425024512 bytes 2024-09-14T05:44:51.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2448111616 bytes 2024-09-14T05:44:51.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2521908224 bytes 2024-09-14T05:44:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2566951936 bytes 2024-09-14T05:44:51.694 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2590536704 bytes 2024-09-14T05:44:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2599641088 bytes 2024-09-14T05:44:51.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2653742080 bytes 2024-09-14T05:44:51.995 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2701489152 bytes 2024-09-14T05:44:52.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2794916864 bytes 2024-09-14T05:44:52.246 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2817062912 bytes 2024-09-14T05:44:52.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 2854181888 bytes 2024-09-14T05:44:52.422 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 2861363200 bytes 2024-09-14T05:44:52.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 2871427072 bytes 2024-09-14T05:44:52.601 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 2894045184 bytes 2024-09-14T05:44:52.689 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 2917197824 bytes 2024-09-14T05:44:52.794 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 2971552768 bytes 2024-09-14T05:44:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 3014791168 bytes 2024-09-14T05:44:53.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7300 IOs, 3043821568 bytes 2024-09-14T05:44:53.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:44:53.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:44:53.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:44:53.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:44:53.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:44:53.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_3 2024-09-14T05:44:53.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_3 snapshot 128M 2024-09-14T05:44:53.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:44:53.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:44:53.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:44:53.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:44:53.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-14T05:44:53.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:44:53.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:44:53.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-14T05:44:53.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:44:53.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_3 128M 2024-09-14T05:44:53.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:44:53.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:44:53.100 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:44:53.100 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:44:53.100 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-14T05:44:53.100 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:44:53.100 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T05:44:53.100 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-14T05:44:53.100 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-14T05:44:53.100 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:44:53.100 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_3 2024-09-14T05:44:53.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T05:44:53.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:44:53.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-14T05:44:53.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:44:53.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_3 snapshot 2024-09-14T05:44:53.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:44:53.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:44:53.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-14T05:44:53.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:44:53.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2024-09-14T05:44:54.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T05:44:54.074 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2024-09-14T05:44:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_3': 2024-09-14T05:44:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:44:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:44:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T05:44:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 127a9f7fe397 2024-09-14T05:44:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.127a9f7fe397 2024-09-14T05:44:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:44:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T05:44:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:44:54.117 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:44:54.117 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:44:53 2024 2024-09-14T05:44:54.117 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:44:53 2024 2024-09-14T05:44:54.117 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:44:53 2024 2024-09-14T05:44:54.117 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:44:54.117 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:44:54.117 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 847da547-970e-41a0-bee0-110873986de0 2024-09-14T05:44:54.117 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T05:44:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:44:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_3 2024-09-14T05:44:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:44:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:44:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-14T05:44:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:44:54.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=5 2024-09-14T05:44:54.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:44:54.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:44:54.291 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 19807232 bytes 2024-09-14T05:44:54.374 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 48212992 bytes 2024-09-14T05:44:54.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 144218112 bytes 2024-09-14T05:44:54.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 198272000 bytes 2024-09-14T05:44:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 245795840 bytes 2024-09-14T05:44:54.836 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 274269184 bytes 2024-09-14T05:44:54.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 347625472 bytes 2024-09-14T05:44:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 356742144 bytes 2024-09-14T05:44:55.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 401601536 bytes 2024-09-14T05:44:55.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 444209152 bytes 2024-09-14T05:44:55.324 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 448457728 bytes 2024-09-14T05:44:55.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 490321920 bytes 2024-09-14T05:44:55.506 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 514543616 bytes 2024-09-14T05:44:55.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 568085504 bytes 2024-09-14T05:44:55.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 630424576 bytes 2024-09-14T05:44:55.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 674037760 bytes 2024-09-14T05:44:55.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 722278400 bytes 2024-09-14T05:44:56.089 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 782560256 bytes 2024-09-14T05:44:56.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 799698944 bytes 2024-09-14T05:44:56.295 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 867877888 bytes 2024-09-14T05:44:56.377 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 868501504 bytes 2024-09-14T05:44:56.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 909254656 bytes 2024-09-14T05:44:56.588 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 944234496 bytes 2024-09-14T05:44:56.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 954980352 bytes 2024-09-14T05:44:56.758 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1011346432 bytes 2024-09-14T05:44:56.930 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1101170688 bytes 2024-09-14T05:44:56.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1101771776 bytes 2024-09-14T05:44:57.089 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1125487616 bytes 2024-09-14T05:44:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1147762688 bytes 2024-09-14T05:44:57.322 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1231245312 bytes 2024-09-14T05:44:57.392 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1231868928 bytes 2024-09-14T05:44:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1261303808 bytes 2024-09-14T05:44:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1328697344 bytes 2024-09-14T05:44:57.739 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1404881920 bytes 2024-09-14T05:44:57.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1447938048 bytes 2024-09-14T05:44:57.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1512629248 bytes 2024-09-14T05:44:58.088 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1549844480 bytes 2024-09-14T05:44:58.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1607460864 bytes 2024-09-14T05:44:58.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1645718528 bytes 2024-09-14T05:44:58.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1695379456 bytes 2024-09-14T05:44:58.548 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1736222720 bytes 2024-09-14T05:44:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1769933824 bytes 2024-09-14T05:44:58.756 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1807102976 bytes 2024-09-14T05:44:58.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1807720448 bytes 2024-09-14T05:44:58.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1846507520 bytes 2024-09-14T05:44:58.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1847150592 bytes 2024-09-14T05:44:59.101 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 1896180736 bytes 2024-09-14T05:44:59.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:44:59.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:44:59.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:44:59.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:44:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:44:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_4 2024-09-14T05:44:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_4 snapshot 128M 2024-09-14T05:44:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:44:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:44:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:44:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:44:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-14T05:44:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:44:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:44:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-14T05:44:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:44:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_4 128M 2024-09-14T05:44:59.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:44:59.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:44:59.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:44:59.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:44:59.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-14T05:44:59.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:44:59.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T05:44:59.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-14T05:44:59.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-14T05:44:59.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:44:59.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_4 2024-09-14T05:44:59.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T05:44:59.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:44:59.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-14T05:44:59.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:44:59.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_4 snapshot 2024-09-14T05:44:59.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:44:59.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:44:59.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-14T05:44:59.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:44:59.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2024-09-14T05:45:00.068 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T05:45:00.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2024-09-14T05:45:00.118 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_4': 2024-09-14T05:45:00.118 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:45:00.118 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:45:00.118 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T05:45:00.118 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 12825e8f94e0 2024-09-14T05:45:00.118 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.12825e8f94e0 2024-09-14T05:45:00.118 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:45:00.118 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T05:45:00.118 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:45:00.118 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:45:00.118 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:44:59 2024 2024-09-14T05:45:00.118 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:44:59 2024 2024-09-14T05:45:00.119 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:44:59 2024 2024-09-14T05:45:00.119 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:45:00.119 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:45:00.119 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 476c0897-664e-4ed7-a165-44f4a9d2c1a5 2024-09-14T05:45:00.119 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T05:45:00.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:45:00.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_4 2024-09-14T05:45:00.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:45:00.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:45:00.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-14T05:45:00.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:45:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=10 2024-09-14T05:45:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:45:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:45:00.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 60594176 bytes 2024-09-14T05:45:00.382 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 75134976 bytes 2024-09-14T05:45:00.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 109897728 bytes 2024-09-14T05:45:00.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 130679808 bytes 2024-09-14T05:45:00.652 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 138520576 bytes 2024-09-14T05:45:00.733 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 161076224 bytes 2024-09-14T05:45:00.835 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 194152448 bytes 2024-09-14T05:45:00.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 268007424 bytes 2024-09-14T05:45:01.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 293368832 bytes 2024-09-14T05:45:01.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 366330880 bytes 2024-09-14T05:45:01.299 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 409619456 bytes 2024-09-14T05:45:01.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 480641024 bytes 2024-09-14T05:45:01.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 496232448 bytes 2024-09-14T05:45:01.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 523977728 bytes 2024-09-14T05:45:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 616910848 bytes 2024-09-14T05:45:01.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 617532416 bytes 2024-09-14T05:45:01.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 634537984 bytes 2024-09-14T05:45:02.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 696826880 bytes 2024-09-14T05:45:02.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 709220352 bytes 2024-09-14T05:45:02.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 721405952 bytes 2024-09-14T05:45:02.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 810087424 bytes 2024-09-14T05:45:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 889278464 bytes 2024-09-14T05:45:02.600 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 918609920 bytes 2024-09-14T05:45:02.681 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 932704256 bytes 2024-09-14T05:45:02.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 962892800 bytes 2024-09-14T05:45:02.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1013822464 bytes 2024-09-14T05:45:02.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1081721856 bytes 2024-09-14T05:45:03.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1131127808 bytes 2024-09-14T05:45:03.250 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1185733632 bytes 2024-09-14T05:45:03.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1257030656 bytes 2024-09-14T05:45:03.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1346784256 bytes 2024-09-14T05:45:03.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1390516224 bytes 2024-09-14T05:45:03.761 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1417059328 bytes 2024-09-14T05:45:03.911 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1508609024 bytes 2024-09-14T05:45:04.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1546733568 bytes 2024-09-14T05:45:04.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1626019840 bytes 2024-09-14T05:45:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1694325760 bytes 2024-09-14T05:45:04.377 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1703346176 bytes 2024-09-14T05:45:04.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1734303744 bytes 2024-09-14T05:45:04.599 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1786717184 bytes 2024-09-14T05:45:04.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1837537280 bytes 2024-09-14T05:45:04.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1858606080 bytes 2024-09-14T05:45:04.911 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1896269824 bytes 2024-09-14T05:45:05.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1941646336 bytes 2024-09-14T05:45:05.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1971695616 bytes 2024-09-14T05:45:05.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1979238400 bytes 2024-09-14T05:45:05.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 2094918656 bytes 2024-09-14T05:45:05.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2127571968 bytes 2024-09-14T05:45:05.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2194335744 bytes 2024-09-14T05:45:05.734 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2239383552 bytes 2024-09-14T05:45:05.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2260034560 bytes 2024-09-14T05:45:05.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2285241344 bytes 2024-09-14T05:45:06.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2303779840 bytes 2024-09-14T05:45:06.160 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2395450368 bytes 2024-09-14T05:45:06.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2415037440 bytes 2024-09-14T05:45:06.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2447409152 bytes 2024-09-14T05:45:06.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2562007040 bytes 2024-09-14T05:45:06.675 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2649135104 bytes 2024-09-14T05:45:06.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2687736832 bytes 2024-09-14T05:45:06.913 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2758232064 bytes 2024-09-14T05:45:07.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2789032960 bytes 2024-09-14T05:45:07.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2795290624 bytes 2024-09-14T05:45:07.198 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2839516160 bytes 2024-09-14T05:45:07.294 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2882189312 bytes 2024-09-14T05:45:07.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2932160512 bytes 2024-09-14T05:45:07.524 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 2953204736 bytes 2024-09-14T05:45:07.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 2966556672 bytes 2024-09-14T05:45:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 3017457664 bytes 2024-09-14T05:45:07.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 3048600576 bytes 2024-09-14T05:45:07.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 3082916864 bytes 2024-09-14T05:45:08.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 3132640256 bytes 2024-09-14T05:45:08.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 3224384512 bytes 2024-09-14T05:45:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7300 IOs, 3240982528 bytes 2024-09-14T05:45:08.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7400 IOs, 3276406784 bytes 2024-09-14T05:45:08.475 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7768246897440497) > weight (0.25) 2024-09-14T05:45:08.475 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-09-14T05:45:08.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-09-14T05:45:08.477 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-09-14T05:45:08.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T05:45:08.476+0000 7fd090f9a640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 36765) UID: 1000 2024-09-14T05:45:08.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7500 IOs, 3314455552 bytes 2024-09-14T05:45:08.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7600 IOs, 3346671616 bytes 2024-09-14T05:45:08.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7700 IOs, 3374343168 bytes 2024-09-14T05:45:08.856 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7800 IOs, 3443673088 bytes 2024-09-14T05:45:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7900 IOs, 3465513984 bytes 2024-09-14T05:45:09.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8000 IOs, 3552287744 bytes 2024-09-14T05:45:09.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8100 IOs, 3584537600 bytes 2024-09-14T05:45:09.333 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8200 IOs, 3641438208 bytes 2024-09-14T05:45:09.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8300 IOs, 3686159360 bytes 2024-09-14T05:45:09.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8400 IOs, 3758170112 bytes 2024-09-14T05:45:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8500 IOs, 3789815808 bytes 2024-09-14T05:45:09.802 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8600 IOs, 3849136128 bytes 2024-09-14T05:45:09.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8700 IOs, 3885313024 bytes 2024-09-14T05:45:09.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8800 IOs, 3904190464 bytes 2024-09-14T05:45:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8900 IOs, 3925192704 bytes 2024-09-14T05:45:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:45:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:45:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:45:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:45:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:45:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_5 2024-09-14T05:45:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_5 snapshot 128M 2024-09-14T05:45:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:45:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:45:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:45:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:45:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-14T05:45:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:45:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:45:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-14T05:45:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:45:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_5 128M 2024-09-14T05:45:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:45:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:45:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:45:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:45:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-14T05:45:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:45:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T05:45:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-14T05:45:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-14T05:45:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:45:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_5 2024-09-14T05:45:10.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T05:45:10.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:45:10.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-14T05:45:10.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:45:10.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_5 snapshot 2024-09-14T05:45:10.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:45:10.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:45:10.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-14T05:45:10.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:45:10.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2024-09-14T05:45:11.089 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T05:45:11.097 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2024-09-14T05:45:11.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_5': 2024-09-14T05:45:11.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:45:11.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:45:11.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T05:45:11.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 128bb28977c7 2024-09-14T05:45:11.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.128bb28977c7 2024-09-14T05:45:11.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:45:11.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T05:45:11.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:45:11.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:45:11.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:45:10 2024 2024-09-14T05:45:11.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:45:10 2024 2024-09-14T05:45:11.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:45:10 2024 2024-09-14T05:45:11.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:45:11.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:45:11.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 5962be87-a0c9-4726-93e1-f3aad0bd9173 2024-09-14T05:45:11.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T05:45:11.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:45:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_5 2024-09-14T05:45:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:45:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:45:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-14T05:45:11.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:45:11.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=10 2024-09-14T05:45:11.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:45:11.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:45:11.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 40740864 bytes 2024-09-14T05:45:11.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 152229888 bytes 2024-09-14T05:45:11.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 176931840 bytes 2024-09-14T05:45:11.696 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 206610432 bytes 2024-09-14T05:45:11.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 237261824 bytes 2024-09-14T05:45:11.911 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 281818112 bytes 2024-09-14T05:45:12.037 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 343396352 bytes 2024-09-14T05:45:12.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 376736768 bytes 2024-09-14T05:45:12.229 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 401524736 bytes 2024-09-14T05:45:12.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 424036352 bytes 2024-09-14T05:45:12.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 474554368 bytes 2024-09-14T05:45:12.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 488742912 bytes 2024-09-14T05:45:12.610 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 508416000 bytes 2024-09-14T05:45:12.681 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 509003776 bytes 2024-09-14T05:45:12.770 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 523607040 bytes 2024-09-14T05:45:12.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 545346560 bytes 2024-09-14T05:45:12.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 599736320 bytes 2024-09-14T05:45:13.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 645553152 bytes 2024-09-14T05:45:13.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 670900224 bytes 2024-09-14T05:45:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 699168768 bytes 2024-09-14T05:45:13.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 736787456 bytes 2024-09-14T05:45:13.558 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 807690240 bytes 2024-09-14T05:45:13.674 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 827100160 bytes 2024-09-14T05:45:13.762 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 842693632 bytes 2024-09-14T05:45:13.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 878273536 bytes 2024-09-14T05:45:13.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 893309952 bytes 2024-09-14T05:45:14.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 923193344 bytes 2024-09-14T05:45:14.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 960885760 bytes 2024-09-14T05:45:14.237 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 977258496 bytes 2024-09-14T05:45:14.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 977886208 bytes 2024-09-14T05:45:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1032992768 bytes 2024-09-14T05:45:14.506 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1059555328 bytes 2024-09-14T05:45:14.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1088941056 bytes 2024-09-14T05:45:14.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1129696256 bytes 2024-09-14T05:45:14.815 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1157525504 bytes 2024-09-14T05:45:14.951 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1230874624 bytes 2024-09-14T05:45:15.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1251174400 bytes 2024-09-14T05:45:15.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1275018240 bytes 2024-09-14T05:45:15.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1290313728 bytes 2024-09-14T05:45:15.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1324193792 bytes 2024-09-14T05:45:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1358746624 bytes 2024-09-14T05:45:15.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1407310848 bytes 2024-09-14T05:45:15.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1431075840 bytes 2024-09-14T05:45:15.744 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1469822976 bytes 2024-09-14T05:45:15.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1531558912 bytes 2024-09-14T05:45:16.084 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1676717056 bytes 2024-09-14T05:45:16.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 1723874304 bytes 2024-09-14T05:45:16.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 1787529216 bytes 2024-09-14T05:45:16.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 1890365440 bytes 2024-09-14T05:45:16.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 1954931712 bytes 2024-09-14T05:45:16.782 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2001091584 bytes 2024-09-14T05:45:16.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2047836160 bytes 2024-09-14T05:45:16.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2063386624 bytes 2024-09-14T05:45:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2069855232 bytes 2024-09-14T05:45:17.132 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2078984192 bytes 2024-09-14T05:45:17.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2111479808 bytes 2024-09-14T05:45:17.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2134466560 bytes 2024-09-14T05:45:17.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2224062464 bytes 2024-09-14T05:45:17.478 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-09-14T05:45:17.478 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T05:45:17.478 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-09-14T05:45:17.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-14T05:45:17.479 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-14T05:45:17.482 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-14T05:45:17.482 INFO:tasks.rbd_mirror_thrash:waiting for 65 secs before thrashing 2024-09-14T05:45:17.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2273979392 bytes 2024-09-14T05:45:17.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2303175680 bytes 2024-09-14T05:45:17.856 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2368975872 bytes 2024-09-14T05:45:17.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2400919552 bytes 2024-09-14T05:45:18.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2416217088 bytes 2024-09-14T05:45:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2478158848 bytes 2024-09-14T05:45:18.246 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2478808064 bytes 2024-09-14T05:45:18.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 2504432640 bytes 2024-09-14T05:45:18.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 2587443200 bytes 2024-09-14T05:45:18.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 2616514560 bytes 2024-09-14T05:45:18.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 2726170624 bytes 2024-09-14T05:45:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 2780645376 bytes 2024-09-14T05:45:18.978 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 2804730880 bytes 2024-09-14T05:45:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 2884321280 bytes 2024-09-14T05:45:19.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7300 IOs, 2979288064 bytes 2024-09-14T05:45:19.402 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7400 IOs, 3034720256 bytes 2024-09-14T05:45:19.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7500 IOs, 3105425408 bytes 2024-09-14T05:45:19.637 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7600 IOs, 3130959872 bytes 2024-09-14T05:45:19.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7700 IOs, 3179153408 bytes 2024-09-14T05:45:19.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7800 IOs, 3203955712 bytes 2024-09-14T05:45:19.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7900 IOs, 3243973632 bytes 2024-09-14T05:45:20.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8000 IOs, 3331658752 bytes 2024-09-14T05:45:20.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8100 IOs, 3357861888 bytes 2024-09-14T05:45:20.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8200 IOs, 3421901824 bytes 2024-09-14T05:45:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8300 IOs, 3422507008 bytes 2024-09-14T05:45:20.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8400 IOs, 3514589184 bytes 2024-09-14T05:45:20.669 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8500 IOs, 3529218048 bytes 2024-09-14T05:45:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8600 IOs, 3571057664 bytes 2024-09-14T05:45:20.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8700 IOs, 3603458048 bytes 2024-09-14T05:45:20.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8800 IOs, 3641293824 bytes 2024-09-14T05:45:21.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8900 IOs, 3674008576 bytes 2024-09-14T05:45:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:45:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:45:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:45:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:45:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:45:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_6 2024-09-14T05:45:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_6 snapshot 128M 2024-09-14T05:45:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:45:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:45:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:45:21.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:45:21.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-14T05:45:21.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:45:21.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:45:21.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-14T05:45:21.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:45:21.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_6 128M 2024-09-14T05:45:21.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:45:21.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:45:21.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:45:21.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:45:21.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-14T05:45:21.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:45:21.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T05:45:21.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-14T05:45:21.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-14T05:45:21.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:45:21.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_6 2024-09-14T05:45:21.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T05:45:21.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:45:21.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-14T05:45:21.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:45:21.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_6 snapshot 2024-09-14T05:45:21.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:45:21.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:45:21.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-14T05:45:21.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:45:21.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2024-09-14T05:45:21.624 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T05:45:21.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2024-09-14T05:45:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_6': 2024-09-14T05:45:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:45:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:45:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T05:45:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 129ffbd440c5 2024-09-14T05:45:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.129ffbd440c5 2024-09-14T05:45:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:45:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T05:45:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:45:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:45:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:45:21 2024 2024-09-14T05:45:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:45:21 2024 2024-09-14T05:45:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:45:21 2024 2024-09-14T05:45:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:45:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:45:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: a86da690-7adb-435f-a4fb-fb58a8d79cb7 2024-09-14T05:45:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T05:45:21.677 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:45:21.677 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_6 2024-09-14T05:45:21.677 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:45:21.677 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:45:21.677 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-14T05:45:21.677 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:45:21.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=5 2024-09-14T05:45:21.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:45:21.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:45:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 44305408 bytes 2024-09-14T05:45:21.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 50242560 bytes 2024-09-14T05:45:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 78289920 bytes 2024-09-14T05:45:22.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 172740608 bytes 2024-09-14T05:45:22.288 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 213274624 bytes 2024-09-14T05:45:22.383 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 224345088 bytes 2024-09-14T05:45:22.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 269124608 bytes 2024-09-14T05:45:22.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 306582528 bytes 2024-09-14T05:45:22.685 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 335031296 bytes 2024-09-14T05:45:22.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 335645696 bytes 2024-09-14T05:45:22.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 392585216 bytes 2024-09-14T05:45:22.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 424345600 bytes 2024-09-14T05:45:23.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 438292480 bytes 2024-09-14T05:45:23.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 486334464 bytes 2024-09-14T05:45:23.292 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 534057984 bytes 2024-09-14T05:45:23.423 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 596306944 bytes 2024-09-14T05:45:23.542 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 645180416 bytes 2024-09-14T05:45:23.692 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 731075584 bytes 2024-09-14T05:45:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 818801664 bytes 2024-09-14T05:45:23.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 863229952 bytes 2024-09-14T05:45:24.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 918543360 bytes 2024-09-14T05:45:24.245 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 1005313024 bytes 2024-09-14T05:45:24.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1057869824 bytes 2024-09-14T05:45:24.459 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1084691456 bytes 2024-09-14T05:45:24.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1181043712 bytes 2024-09-14T05:45:24.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1199753216 bytes 2024-09-14T05:45:24.781 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1210358784 bytes 2024-09-14T05:45:24.865 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1213424640 bytes 2024-09-14T05:45:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1250013184 bytes 2024-09-14T05:45:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1289724928 bytes 2024-09-14T05:45:25.190 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1340209152 bytes 2024-09-14T05:45:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1360315392 bytes 2024-09-14T05:45:25.368 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1381542912 bytes 2024-09-14T05:45:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1426100224 bytes 2024-09-14T05:45:25.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1455850496 bytes 2024-09-14T05:45:25.694 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1495162880 bytes 2024-09-14T05:45:25.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1495772160 bytes 2024-09-14T05:45:25.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1516006400 bytes 2024-09-14T05:45:25.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1530607616 bytes 2024-09-14T05:45:26.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1606818816 bytes 2024-09-14T05:45:26.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1641350144 bytes 2024-09-14T05:45:26.286 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1669124096 bytes 2024-09-14T05:45:26.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1705280512 bytes 2024-09-14T05:45:26.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1733999616 bytes 2024-09-14T05:45:26.604 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1785433088 bytes 2024-09-14T05:45:26.686 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:45:26.686 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:45:26.686 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:45:26.686 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:45:26.686 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:45:26.686 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_7 2024-09-14T05:45:26.686 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_7 snapshot 128M 2024-09-14T05:45:26.686 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:45:26.686 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:45:26.686 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:45:26.686 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:45:26.687 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-14T05:45:26.687 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:45:26.687 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:45:26.688 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-14T05:45:26.688 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:45:26.688 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_7 128M 2024-09-14T05:45:26.688 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:45:26.688 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:45:26.688 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:45:26.688 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:45:26.688 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-14T05:45:26.689 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:45:26.689 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T05:45:26.689 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-14T05:45:26.689 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-14T05:45:26.689 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:45:26.689 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_7 2024-09-14T05:45:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T05:45:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:45:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-14T05:45:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:45:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_7 snapshot 2024-09-14T05:45:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:45:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:45:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-14T05:45:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:45:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2024-09-14T05:45:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T05:45:27.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2024-09-14T05:45:27.689 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_7': 2024-09-14T05:45:27.689 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:45:27.689 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:45:27.689 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T05:45:27.689 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 12ab84c61cc1 2024-09-14T05:45:27.689 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.12ab84c61cc1 2024-09-14T05:45:27.689 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:45:27.689 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T05:45:27.690 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:45:27.690 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:45:27.690 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:45:26 2024 2024-09-14T05:45:27.690 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:45:26 2024 2024-09-14T05:45:27.690 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:45:26 2024 2024-09-14T05:45:27.690 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:45:27.690 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:45:27.690 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 34d473d8-f5ad-4be8-be0a-057254b091f7 2024-09-14T05:45:27.690 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T05:45:27.693 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:45:27.693 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_7 2024-09-14T05:45:27.693 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:45:27.693 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:45:27.693 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-14T05:45:27.693 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:45:27.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=12 2024-09-14T05:45:27.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:45:27.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:45:27.836 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 630784 bytes 2024-09-14T05:45:27.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 18254848 bytes 2024-09-14T05:45:28.014 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 44992512 bytes 2024-09-14T05:45:28.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 113446912 bytes 2024-09-14T05:45:28.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 162441216 bytes 2024-09-14T05:45:28.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 189980672 bytes 2024-09-14T05:45:28.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 309354496 bytes 2024-09-14T05:45:28.698 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 345280512 bytes 2024-09-14T05:45:28.851 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 438067200 bytes 2024-09-14T05:45:28.930 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 447154176 bytes 2024-09-14T05:45:29.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 512935936 bytes 2024-09-14T05:45:29.220 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 614621184 bytes 2024-09-14T05:45:29.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 639040512 bytes 2024-09-14T05:45:29.393 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 664603648 bytes 2024-09-14T05:45:29.468 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 679539712 bytes 2024-09-14T05:45:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 748971008 bytes 2024-09-14T05:45:29.675 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 749581312 bytes 2024-09-14T05:45:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 780215296 bytes 2024-09-14T05:45:29.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 814521344 bytes 2024-09-14T05:45:29.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 839858176 bytes 2024-09-14T05:45:30.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 861640704 bytes 2024-09-14T05:45:30.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 931193856 bytes 2024-09-14T05:45:30.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 990296064 bytes 2024-09-14T05:45:30.392 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1009688576 bytes 2024-09-14T05:45:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1040670720 bytes 2024-09-14T05:45:30.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1087110144 bytes 2024-09-14T05:45:30.689 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1108901888 bytes 2024-09-14T05:45:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1146092544 bytes 2024-09-14T05:45:30.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1203696640 bytes 2024-09-14T05:45:31.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1232587776 bytes 2024-09-14T05:45:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1281548288 bytes 2024-09-14T05:45:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1334930432 bytes 2024-09-14T05:45:31.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1473150976 bytes 2024-09-14T05:45:31.506 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1499113472 bytes 2024-09-14T05:45:31.637 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1567259648 bytes 2024-09-14T05:45:31.749 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1596579840 bytes 2024-09-14T05:45:31.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1667254272 bytes 2024-09-14T05:45:32.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1712579584 bytes 2024-09-14T05:45:32.132 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1771547648 bytes 2024-09-14T05:45:32.227 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1794083840 bytes 2024-09-14T05:45:32.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1835736064 bytes 2024-09-14T05:45:32.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1860636672 bytes 2024-09-14T05:45:32.534 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1890516992 bytes 2024-09-14T05:45:32.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1925745664 bytes 2024-09-14T05:45:32.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1926378496 bytes 2024-09-14T05:45:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1961508864 bytes 2024-09-14T05:45:32.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 2016390144 bytes 2024-09-14T05:45:33.038 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2050410496 bytes 2024-09-14T05:45:33.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2073809920 bytes 2024-09-14T05:45:33.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2159712256 bytes 2024-09-14T05:45:33.374 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2197706752 bytes 2024-09-14T05:45:33.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2233027584 bytes 2024-09-14T05:45:33.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2258336768 bytes 2024-09-14T05:45:33.691 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2298634240 bytes 2024-09-14T05:45:33.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2384516096 bytes 2024-09-14T05:45:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2426789888 bytes 2024-09-14T05:45:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2511171584 bytes 2024-09-14T05:45:34.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2531174400 bytes 2024-09-14T05:45:34.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2532094976 bytes 2024-09-14T05:45:34.387 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2569712640 bytes 2024-09-14T05:45:34.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2606537728 bytes 2024-09-14T05:45:34.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2638444544 bytes 2024-09-14T05:45:34.687 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2661245952 bytes 2024-09-14T05:45:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2694532096 bytes 2024-09-14T05:45:34.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2721728512 bytes 2024-09-14T05:45:35.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 2771398656 bytes 2024-09-14T05:45:35.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 2771993600 bytes 2024-09-14T05:45:35.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 2797200384 bytes 2024-09-14T05:45:35.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 2815105024 bytes 2024-09-14T05:45:35.426 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 2909796352 bytes 2024-09-14T05:45:35.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 2915826688 bytes 2024-09-14T05:45:35.637 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 2979963904 bytes 2024-09-14T05:45:35.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7300 IOs, 3029455872 bytes 2024-09-14T05:45:35.855 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7400 IOs, 3059942400 bytes 2024-09-14T05:45:35.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7500 IOs, 3091462144 bytes 2024-09-14T05:45:36.070 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7600 IOs, 3134336000 bytes 2024-09-14T05:45:36.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7700 IOs, 3159286784 bytes 2024-09-14T05:45:36.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7800 IOs, 3245949952 bytes 2024-09-14T05:45:36.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7900 IOs, 3315777536 bytes 2024-09-14T05:45:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8000 IOs, 3358910464 bytes 2024-09-14T05:45:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8100 IOs, 3380304896 bytes 2024-09-14T05:45:36.751 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8200 IOs, 3397348352 bytes 2024-09-14T05:45:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8300 IOs, 3412056064 bytes 2024-09-14T05:45:36.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8400 IOs, 3481060352 bytes 2024-09-14T05:45:37.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8500 IOs, 3487010816 bytes 2024-09-14T05:45:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8600 IOs, 3510888448 bytes 2024-09-14T05:45:37.270 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8700 IOs, 3566216192 bytes 2024-09-14T05:45:37.350 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8800 IOs, 3573900288 bytes 2024-09-14T05:45:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8900 IOs, 3632021504 bytes 2024-09-14T05:45:37.613 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9000 IOs, 3709671424 bytes 2024-09-14T05:45:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9100 IOs, 3749978112 bytes 2024-09-14T05:45:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9200 IOs, 3870465024 bytes 2024-09-14T05:45:38.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9300 IOs, 3884266496 bytes 2024-09-14T05:45:38.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9400 IOs, 3929738240 bytes 2024-09-14T05:45:38.229 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9500 IOs, 3974038528 bytes 2024-09-14T05:45:38.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9600 IOs, 4022963200 bytes 2024-09-14T05:45:38.455 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9700 IOs, 4063350784 bytes 2024-09-14T05:45:38.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9800 IOs, 4063994880 bytes 2024-09-14T05:45:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9900 IOs, 4137285632 bytes 2024-09-14T05:45:38.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10000 IOs, 4155400192 bytes 2024-09-14T05:45:38.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10100 IOs, 4240618496 bytes 2024-09-14T05:45:39.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10200 IOs, 4317723648 bytes 2024-09-14T05:45:39.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10300 IOs, 4386868224 bytes 2024-09-14T05:45:39.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10400 IOs, 4416124928 bytes 2024-09-14T05:45:39.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10500 IOs, 4477322240 bytes 2024-09-14T05:45:39.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10600 IOs, 4530924544 bytes 2024-09-14T05:45:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10700 IOs, 4539295744 bytes 2024-09-14T05:45:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:45:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:45:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:45:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:45:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:45:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_8 2024-09-14T05:45:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_8 snapshot 128M 2024-09-14T05:45:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:45:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:45:39.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:45:39.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:45:39.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-14T05:45:39.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:45:39.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:45:39.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-14T05:45:39.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:45:39.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_8 128M 2024-09-14T05:45:39.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:45:39.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:45:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:45:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:45:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-14T05:45:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:45:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T05:45:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-14T05:45:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-14T05:45:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:45:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_8 2024-09-14T05:45:39.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T05:45:39.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:45:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-14T05:45:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:45:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_8 snapshot 2024-09-14T05:45:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:45:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:45:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-14T05:45:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:45:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2024-09-14T05:45:40.658 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T05:45:40.668 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2024-09-14T05:45:40.710 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_8': 2024-09-14T05:45:40.711 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:45:40.711 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:45:40.711 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T05:45:40.711 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 12b77f8e016d 2024-09-14T05:45:40.711 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.12b77f8e016d 2024-09-14T05:45:40.711 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:45:40.711 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T05:45:40.711 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:45:40.711 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:45:40.711 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:45:39 2024 2024-09-14T05:45:40.711 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:45:39 2024 2024-09-14T05:45:40.711 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:45:39 2024 2024-09-14T05:45:40.712 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:45:40.712 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:45:40.712 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 21958cb0-e88b-48db-bd17-5fa7463f50e0 2024-09-14T05:45:40.712 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T05:45:40.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:45:40.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_8 2024-09-14T05:45:40.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:45:40.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:45:40.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-14T05:45:40.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:45:40.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=9 2024-09-14T05:45:40.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:45:40.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:45:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 607232 bytes 2024-09-14T05:45:40.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 45824000 bytes 2024-09-14T05:45:41.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 84714496 bytes 2024-09-14T05:45:41.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 123312128 bytes 2024-09-14T05:45:41.287 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 146987008 bytes 2024-09-14T05:45:41.380 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 176676864 bytes 2024-09-14T05:45:41.489 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 215864320 bytes 2024-09-14T05:45:41.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 275938304 bytes 2024-09-14T05:45:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 345328640 bytes 2024-09-14T05:45:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 347612160 bytes 2024-09-14T05:45:41.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 379920384 bytes 2024-09-14T05:45:42.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 399080448 bytes 2024-09-14T05:45:42.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 476264448 bytes 2024-09-14T05:45:42.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 529274880 bytes 2024-09-14T05:45:42.382 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 558486528 bytes 2024-09-14T05:45:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 610012160 bytes 2024-09-14T05:45:42.577 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 619591680 bytes 2024-09-14T05:45:42.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 659958784 bytes 2024-09-14T05:45:42.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 706550784 bytes 2024-09-14T05:45:42.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 749742080 bytes 2024-09-14T05:45:43.021 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 762162176 bytes 2024-09-14T05:45:43.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 792811520 bytes 2024-09-14T05:45:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 864645120 bytes 2024-09-14T05:45:43.393 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 923373568 bytes 2024-09-14T05:45:43.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 975707136 bytes 2024-09-14T05:45:43.638 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1016067072 bytes 2024-09-14T05:45:43.731 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1039351808 bytes 2024-09-14T05:45:43.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1059680256 bytes 2024-09-14T05:45:43.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1078711296 bytes 2024-09-14T05:45:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1079301120 bytes 2024-09-14T05:45:44.094 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1105739776 bytes 2024-09-14T05:45:44.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1115486208 bytes 2024-09-14T05:45:44.340 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1208215552 bytes 2024-09-14T05:45:44.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1255288832 bytes 2024-09-14T05:45:44.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1312863232 bytes 2024-09-14T05:45:44.689 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1352227840 bytes 2024-09-14T05:45:44.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1394859008 bytes 2024-09-14T05:45:44.929 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1446505472 bytes 2024-09-14T05:45:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1504926720 bytes 2024-09-14T05:45:45.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1505531904 bytes 2024-09-14T05:45:45.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1532460032 bytes 2024-09-14T05:45:45.352 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1585081344 bytes 2024-09-14T05:45:45.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1605726208 bytes 2024-09-14T05:45:45.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1649417216 bytes 2024-09-14T05:45:45.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1694135296 bytes 2024-09-14T05:45:45.771 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1713453056 bytes 2024-09-14T05:45:45.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 1732872192 bytes 2024-09-14T05:45:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 1763108864 bytes 2024-09-14T05:45:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 1789484032 bytes 2024-09-14T05:45:46.161 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 1812468736 bytes 2024-09-14T05:45:46.287 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 1893147648 bytes 2024-09-14T05:45:46.460 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 1974238208 bytes 2024-09-14T05:45:46.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 1977812992 bytes 2024-09-14T05:45:46.676 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2066324480 bytes 2024-09-14T05:45:46.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2119566336 bytes 2024-09-14T05:45:46.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2201312256 bytes 2024-09-14T05:45:47.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2201924608 bytes 2024-09-14T05:45:47.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2226741248 bytes 2024-09-14T05:45:47.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2247440384 bytes 2024-09-14T05:45:47.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2289552384 bytes 2024-09-14T05:45:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2346449920 bytes 2024-09-14T05:45:47.618 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2419919872 bytes 2024-09-14T05:45:47.694 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2420936704 bytes 2024-09-14T05:45:47.811 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2465718272 bytes 2024-09-14T05:45:47.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2491475968 bytes 2024-09-14T05:45:48.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 2531241984 bytes 2024-09-14T05:45:48.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 2577339392 bytes 2024-09-14T05:45:48.221 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 2577938432 bytes 2024-09-14T05:45:48.352 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 2636127232 bytes 2024-09-14T05:45:48.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 2712339456 bytes 2024-09-14T05:45:48.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 2726671360 bytes 2024-09-14T05:45:48.700 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 2765589504 bytes 2024-09-14T05:45:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7300 IOs, 2822230016 bytes 2024-09-14T05:45:48.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7400 IOs, 2838300672 bytes 2024-09-14T05:45:49.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7500 IOs, 2874693632 bytes 2024-09-14T05:45:49.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7600 IOs, 2898404352 bytes 2024-09-14T05:45:49.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7700 IOs, 2917532672 bytes 2024-09-14T05:45:49.325 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7800 IOs, 2944964608 bytes 2024-09-14T05:45:49.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7900 IOs, 2965949440 bytes 2024-09-14T05:45:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8000 IOs, 2989584384 bytes 2024-09-14T05:45:49.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8100 IOs, 3036414976 bytes 2024-09-14T05:45:49.722 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:45:49.722 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:45:49.722 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:45:49.722 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:45:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:45:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_9 2024-09-14T05:45:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_9 snapshot 128M 2024-09-14T05:45:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:45:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:45:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:45:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:45:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-14T05:45:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:45:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:45:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-14T05:45:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:45:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_9 128M 2024-09-14T05:45:49.724 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:45:49.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:45:49.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:45:49.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:45:49.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-14T05:45:49.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:45:49.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T05:45:49.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-14T05:45:49.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-14T05:45:49.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:45:49.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_9 2024-09-14T05:45:49.773 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T05:45:49.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:45:49.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-14T05:45:49.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:45:49.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_9 snapshot 2024-09-14T05:45:49.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:45:49.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:45:49.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-14T05:45:49.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:45:49.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2024-09-14T05:45:50.674 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T05:45:50.684 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2024-09-14T05:45:50.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_9': 2024-09-14T05:45:50.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:45:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:45:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T05:45:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 12c32cf43828 2024-09-14T05:45:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.12c32cf43828 2024-09-14T05:45:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:45:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T05:45:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:45:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:45:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:45:49 2024 2024-09-14T05:45:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:45:49 2024 2024-09-14T05:45:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:45:49 2024 2024-09-14T05:45:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:45:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:45:50.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 8d3c3903-631c-42c7-8d30-a5b307b883eb 2024-09-14T05:45:50.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T05:45:50.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:45:50.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_9 2024-09-14T05:45:50.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:45:50.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:45:50.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-14T05:45:50.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:45:50.731 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=12 2024-09-14T05:45:50.731 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:45:50.731 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:45:50.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 3549184 bytes 2024-09-14T05:45:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 81052672 bytes 2024-09-14T05:45:51.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 106337280 bytes 2024-09-14T05:45:51.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 196308992 bytes 2024-09-14T05:45:51.426 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 301925376 bytes 2024-09-14T05:45:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 360708096 bytes 2024-09-14T05:45:51.677 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 424835072 bytes 2024-09-14T05:45:51.796 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 471116800 bytes 2024-09-14T05:45:51.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 518436864 bytes 2024-09-14T05:45:52.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 637415424 bytes 2024-09-14T05:45:52.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 680913920 bytes 2024-09-14T05:45:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 681565184 bytes 2024-09-14T05:45:52.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 692538368 bytes 2024-09-14T05:45:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 729528320 bytes 2024-09-14T05:45:52.588 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 775376896 bytes 2024-09-14T05:45:52.693 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 811111424 bytes 2024-09-14T05:45:52.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 815581184 bytes 2024-09-14T05:45:52.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 865031168 bytes 2024-09-14T05:45:53.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 934470656 bytes 2024-09-14T05:45:53.101 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 937675776 bytes 2024-09-14T05:45:53.193 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 972748800 bytes 2024-09-14T05:45:53.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 1021442048 bytes 2024-09-14T05:45:53.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1045869568 bytes 2024-09-14T05:45:53.745 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1180490752 bytes 2024-09-14T05:45:53.889 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1281021952 bytes 2024-09-14T05:45:54.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1297698816 bytes 2024-09-14T05:45:54.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1338301440 bytes 2024-09-14T05:45:54.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1370974208 bytes 2024-09-14T05:45:54.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1384177664 bytes 2024-09-14T05:45:54.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1425232896 bytes 2024-09-14T05:45:54.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1479691264 bytes 2024-09-14T05:45:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1559271424 bytes 2024-09-14T05:45:54.787 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1605627904 bytes 2024-09-14T05:45:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1634450432 bytes 2024-09-14T05:45:55.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1719727104 bytes 2024-09-14T05:45:55.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1797408768 bytes 2024-09-14T05:45:55.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1814656000 bytes 2024-09-14T05:45:55.421 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1892911104 bytes 2024-09-14T05:45:55.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 2004780032 bytes 2024-09-14T05:45:55.705 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 2037038080 bytes 2024-09-14T05:45:55.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 2081099776 bytes 2024-09-14T05:45:55.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 2105157632 bytes 2024-09-14T05:45:55.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 2105772032 bytes 2024-09-14T05:45:56.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 2131860480 bytes 2024-09-14T05:45:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 2166176768 bytes 2024-09-14T05:45:56.328 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 2228618240 bytes 2024-09-14T05:45:56.452 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 2283909120 bytes 2024-09-14T05:45:56.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2343470080 bytes 2024-09-14T05:45:56.708 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2398730240 bytes 2024-09-14T05:45:56.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2428867584 bytes 2024-09-14T05:45:56.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2467005440 bytes 2024-09-14T05:45:57.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2527892480 bytes 2024-09-14T05:45:57.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2556139520 bytes 2024-09-14T05:45:57.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2588829696 bytes 2024-09-14T05:45:57.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2592069632 bytes 2024-09-14T05:45:57.407 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2594211840 bytes 2024-09-14T05:45:57.462 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9420988806013174) > weight (0.25) 2024-09-14T05:45:57.462 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7700204595281133) > weight (0.25) 2024-09-14T05:45:57.462 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.7810384351644608) > weight (0.25) 2024-09-14T05:45:57.462 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.28248240404678693) > weight (0.25) 2024-09-14T05:45:57.462 INFO:tasks.rbd_mirror_thrash:waiting for 95 secs before thrashing 2024-09-14T05:45:57.529 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2646590464 bytes 2024-09-14T05:45:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2673582080 bytes 2024-09-14T05:45:57.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2745839616 bytes 2024-09-14T05:45:57.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2792709120 bytes 2024-09-14T05:45:57.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2813619200 bytes 2024-09-14T05:45:58.084 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2849005568 bytes 2024-09-14T05:45:58.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2914646016 bytes 2024-09-14T05:45:58.329 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2950440960 bytes 2024-09-14T05:45:58.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 3000258560 bytes 2024-09-14T05:45:58.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 3034689536 bytes 2024-09-14T05:45:58.691 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 3097919488 bytes 2024-09-14T05:45:58.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 3116334080 bytes 2024-09-14T05:45:58.864 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 3131969536 bytes 2024-09-14T05:45:58.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 3157324800 bytes 2024-09-14T05:45:59.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 3157947392 bytes 2024-09-14T05:45:59.157 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 3196307456 bytes 2024-09-14T05:45:59.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7300 IOs, 3251435520 bytes 2024-09-14T05:45:59.383 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7400 IOs, 3282203648 bytes 2024-09-14T05:45:59.464 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7500 IOs, 3291982848 bytes 2024-09-14T05:45:59.564 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7600 IOs, 3325393920 bytes 2024-09-14T05:45:59.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7700 IOs, 3337627648 bytes 2024-09-14T05:45:59.748 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7800 IOs, 3361045504 bytes 2024-09-14T05:45:59.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7900 IOs, 3393378304 bytes 2024-09-14T05:45:59.931 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8000 IOs, 3406544896 bytes 2024-09-14T05:46:00.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8100 IOs, 3430419456 bytes 2024-09-14T05:46:00.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8200 IOs, 3431049216 bytes 2024-09-14T05:46:00.225 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8300 IOs, 3485978624 bytes 2024-09-14T05:46:00.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8400 IOs, 3529734144 bytes 2024-09-14T05:46:00.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8500 IOs, 3575732224 bytes 2024-09-14T05:46:00.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8600 IOs, 3586489344 bytes 2024-09-14T05:46:00.656 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8700 IOs, 3618086912 bytes 2024-09-14T05:46:00.745 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8800 IOs, 3637297152 bytes 2024-09-14T05:46:00.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8900 IOs, 3695325184 bytes 2024-09-14T05:46:00.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9000 IOs, 3722366976 bytes 2024-09-14T05:46:01.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9100 IOs, 3722997760 bytes 2024-09-14T05:46:01.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9200 IOs, 3789865984 bytes 2024-09-14T05:46:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9300 IOs, 3798276096 bytes 2024-09-14T05:46:01.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9400 IOs, 3834454016 bytes 2024-09-14T05:46:01.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9500 IOs, 3850929152 bytes 2024-09-14T05:46:01.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9600 IOs, 3852258304 bytes 2024-09-14T05:46:01.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9700 IOs, 3916884992 bytes 2024-09-14T05:46:01.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9800 IOs, 3975323648 bytes 2024-09-14T05:46:01.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9900 IOs, 4004278272 bytes 2024-09-14T05:46:02.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10000 IOs, 4025406464 bytes 2024-09-14T05:46:02.097 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10100 IOs, 4040955904 bytes 2024-09-14T05:46:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10200 IOs, 4048937984 bytes 2024-09-14T05:46:02.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10300 IOs, 4121327616 bytes 2024-09-14T05:46:02.436 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10400 IOs, 4155565056 bytes 2024-09-14T05:46:02.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10500 IOs, 4182875136 bytes 2024-09-14T05:46:02.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10600 IOs, 4216423424 bytes 2024-09-14T05:46:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:46:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:46:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:46:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:46:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:46:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_10 2024-09-14T05:46:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_10 snapshot 128M 2024-09-14T05:46:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:46:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:46:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:46:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:46:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-14T05:46:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:46:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:46:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-14T05:46:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:46:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_10 128M 2024-09-14T05:46:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:46:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:46:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:46:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:46:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-14T05:46:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:46:02.740 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T05:46:02.740 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-14T05:46:02.740 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-14T05:46:02.740 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:46:02.740 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_10 2024-09-14T05:46:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T05:46:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:46:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-14T05:46:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:46:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_10 snapshot 2024-09-14T05:46:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:46:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:46:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-14T05:46:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:46:02.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2024-09-14T05:46:03.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T05:46:03.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2024-09-14T05:46:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_10': 2024-09-14T05:46:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:46:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:46:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T05:46:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 12cf8ede7423 2024-09-14T05:46:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.12cf8ede7423 2024-09-14T05:46:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:46:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T05:46:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:46:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:46:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:46:02 2024 2024-09-14T05:46:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:46:02 2024 2024-09-14T05:46:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:46:02 2024 2024-09-14T05:46:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:46:03.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:46:03.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: c0427ec2-f759-4269-ac43-364d5f6678fa 2024-09-14T05:46:03.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T05:46:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:46:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_10 2024-09-14T05:46:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:46:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:46:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-14T05:46:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:46:03.761 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=9 2024-09-14T05:46:03.761 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:46:03.761 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:46:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 63723520 bytes 2024-09-14T05:46:04.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 106766336 bytes 2024-09-14T05:46:04.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 141418496 bytes 2024-09-14T05:46:04.322 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 193090560 bytes 2024-09-14T05:46:04.436 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 232251392 bytes 2024-09-14T05:46:04.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 245714944 bytes 2024-09-14T05:46:04.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 290123776 bytes 2024-09-14T05:46:04.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 409379840 bytes 2024-09-14T05:46:04.929 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 437999616 bytes 2024-09-14T05:46:05.040 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 474878976 bytes 2024-09-14T05:46:05.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 496415744 bytes 2024-09-14T05:46:05.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 530897920 bytes 2024-09-14T05:46:05.377 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 593058816 bytes 2024-09-14T05:46:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 622210048 bytes 2024-09-14T05:46:05.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 664116224 bytes 2024-09-14T05:46:05.708 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 716976128 bytes 2024-09-14T05:46:05.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 760368128 bytes 2024-09-14T05:46:05.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 847817728 bytes 2024-09-14T05:46:06.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 876270592 bytes 2024-09-14T05:46:06.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 876872704 bytes 2024-09-14T05:46:06.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 942779392 bytes 2024-09-14T05:46:06.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 1023460352 bytes 2024-09-14T05:46:06.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1081479168 bytes 2024-09-14T05:46:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1105363968 bytes 2024-09-14T05:46:06.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1139478528 bytes 2024-09-14T05:46:06.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1195254784 bytes 2024-09-14T05:46:07.037 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1234992128 bytes 2024-09-14T05:46:07.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1235646464 bytes 2024-09-14T05:46:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1301471232 bytes 2024-09-14T05:46:07.387 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1356206080 bytes 2024-09-14T05:46:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1407670272 bytes 2024-09-14T05:46:07.612 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1432540160 bytes 2024-09-14T05:46:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1468809216 bytes 2024-09-14T05:46:07.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1489761280 bytes 2024-09-14T05:46:07.899 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1504990208 bytes 2024-09-14T05:46:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1505596416 bytes 2024-09-14T05:46:08.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1562211328 bytes 2024-09-14T05:46:08.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1634856960 bytes 2024-09-14T05:46:08.383 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1658483712 bytes 2024-09-14T05:46:08.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1706857472 bytes 2024-09-14T05:46:08.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1721705472 bytes 2024-09-14T05:46:08.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1863428096 bytes 2024-09-14T05:46:08.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1945413632 bytes 2024-09-14T05:46:09.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1985543168 bytes 2024-09-14T05:46:09.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 2050749440 bytes 2024-09-14T05:46:09.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 2109118464 bytes 2024-09-14T05:46:09.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 2197681152 bytes 2024-09-14T05:46:09.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2261463040 bytes 2024-09-14T05:46:09.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2357203968 bytes 2024-09-14T05:46:10.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2418757632 bytes 2024-09-14T05:46:10.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2501975040 bytes 2024-09-14T05:46:10.247 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2514599936 bytes 2024-09-14T05:46:10.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2578764800 bytes 2024-09-14T05:46:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2621334528 bytes 2024-09-14T05:46:10.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2678145024 bytes 2024-09-14T05:46:10.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2691832832 bytes 2024-09-14T05:46:10.812 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2703661056 bytes 2024-09-14T05:46:10.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2721284096 bytes 2024-09-14T05:46:11.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2742268928 bytes 2024-09-14T05:46:11.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2843347968 bytes 2024-09-14T05:46:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2881966080 bytes 2024-09-14T05:46:11.373 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2898486272 bytes 2024-09-14T05:46:11.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2945926144 bytes 2024-09-14T05:46:11.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2991559680 bytes 2024-09-14T05:46:11.765 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 3070791680 bytes 2024-09-14T05:46:11.852 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 3087682560 bytes 2024-09-14T05:46:11.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 3118488576 bytes 2024-09-14T05:46:12.089 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 3184034816 bytes 2024-09-14T05:46:12.220 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 3251462144 bytes 2024-09-14T05:46:12.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 3297223680 bytes 2024-09-14T05:46:12.453 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 3335141376 bytes 2024-09-14T05:46:12.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 3404006400 bytes 2024-09-14T05:46:12.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7300 IOs, 3459798016 bytes 2024-09-14T05:46:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:46:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:46:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:46:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:46:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:46:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_11 2024-09-14T05:46:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_11 snapshot 128M 2024-09-14T05:46:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:46:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:46:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:46:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:46:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_11 2024-09-14T05:46:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:46:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:46:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-14T05:46:12.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:46:12.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_11 128M 2024-09-14T05:46:12.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:46:12.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:46:12.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:46:12.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:46:12.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_11 2024-09-14T05:46:12.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:46:12.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T05:46:12.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-14T05:46:12.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-14T05:46:12.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:46:12.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_11 2024-09-14T05:46:12.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T05:46:12.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:46:12.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-14T05:46:12.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:46:12.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_11 snapshot 2024-09-14T05:46:12.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:46:12.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:46:12.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_11 2024-09-14T05:46:12.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:46:12.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_11 snapshot 2024-09-14T05:46:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T05:46:13.731 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_11 2024-09-14T05:46:13.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_11': 2024-09-14T05:46:13.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:46:13.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:46:13.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T05:46:13.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 12dbb389f805 2024-09-14T05:46:13.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.12dbb389f805 2024-09-14T05:46:13.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:46:13.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T05:46:13.773 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:46:13.773 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:46:13.773 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:46:12 2024 2024-09-14T05:46:13.773 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:46:12 2024 2024-09-14T05:46:13.773 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:46:12 2024 2024-09-14T05:46:13.773 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:46:13.773 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:46:13.773 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: b6cea04e-7839-4e62-8e00-34e757bf5550 2024-09-14T05:46:13.773 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T05:46:13.776 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:46:13.776 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_11 2024-09-14T05:46:13.776 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:46:13.776 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:46:13.776 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_11 2024-09-14T05:46:13.776 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:46:13.778 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=12 2024-09-14T05:46:13.778 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:46:13.778 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_11 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:46:13.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 70006784 bytes 2024-09-14T05:46:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 114700288 bytes 2024-09-14T05:46:14.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 155028480 bytes 2024-09-14T05:46:14.326 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 180967424 bytes 2024-09-14T05:46:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 185613312 bytes 2024-09-14T05:46:14.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 254668800 bytes 2024-09-14T05:46:14.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 265289728 bytes 2024-09-14T05:46:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 334254080 bytes 2024-09-14T05:46:14.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 362254336 bytes 2024-09-14T05:46:15.051 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 458446848 bytes 2024-09-14T05:46:15.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 511259648 bytes 2024-09-14T05:46:15.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 561808384 bytes 2024-09-14T05:46:15.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 620076032 bytes 2024-09-14T05:46:15.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 620711936 bytes 2024-09-14T05:46:15.601 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 621314048 bytes 2024-09-14T05:46:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 659260416 bytes 2024-09-14T05:46:15.856 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 722867200 bytes 2024-09-14T05:46:16.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 831367168 bytes 2024-09-14T05:46:16.221 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 921223168 bytes 2024-09-14T05:46:16.341 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 971053056 bytes 2024-09-14T05:46:16.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 999520256 bytes 2024-09-14T05:46:16.538 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 1019915264 bytes 2024-09-14T05:46:16.709 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1110916096 bytes 2024-09-14T05:46:16.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1169299456 bytes 2024-09-14T05:46:16.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1217544192 bytes 2024-09-14T05:46:17.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1222391808 bytes 2024-09-14T05:46:17.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1240900608 bytes 2024-09-14T05:46:17.292 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1316578304 bytes 2024-09-14T05:46:17.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1340043264 bytes 2024-09-14T05:46:17.532 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1405178880 bytes 2024-09-14T05:46:17.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1435834368 bytes 2024-09-14T05:46:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1455357952 bytes 2024-09-14T05:46:17.859 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1497916416 bytes 2024-09-14T05:46:17.978 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1541654528 bytes 2024-09-14T05:46:18.089 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1572146176 bytes 2024-09-14T05:46:18.193 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1601515520 bytes 2024-09-14T05:46:18.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1668923392 bytes 2024-09-14T05:46:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1740244992 bytes 2024-09-14T05:46:18.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1755955200 bytes 2024-09-14T05:46:18.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1810579456 bytes 2024-09-14T05:46:18.856 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1876246528 bytes 2024-09-14T05:46:18.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1932780544 bytes 2024-09-14T05:46:19.095 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1964306432 bytes 2024-09-14T05:46:19.221 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 2017579008 bytes 2024-09-14T05:46:19.421 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 2145044480 bytes 2024-09-14T05:46:19.578 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 2222674944 bytes 2024-09-14T05:46:19.693 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 2254871552 bytes 2024-09-14T05:46:19.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2303995904 bytes 2024-09-14T05:46:19.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2347333632 bytes 2024-09-14T05:46:20.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2383897600 bytes 2024-09-14T05:46:20.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2401674240 bytes 2024-09-14T05:46:20.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2422883328 bytes 2024-09-14T05:46:20.374 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2477575168 bytes 2024-09-14T05:46:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2519500800 bytes 2024-09-14T05:46:20.626 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2581131264 bytes 2024-09-14T05:46:20.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2645491712 bytes 2024-09-14T05:46:20.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2711277568 bytes 2024-09-14T05:46:21.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2759617536 bytes 2024-09-14T05:46:21.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2783936512 bytes 2024-09-14T05:46:21.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2810526720 bytes 2024-09-14T05:46:21.402 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2901234688 bytes 2024-09-14T05:46:21.528 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2928816128 bytes 2024-09-14T05:46:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 3036279808 bytes 2024-09-14T05:46:21.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 3114283008 bytes 2024-09-14T05:46:21.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 3139644416 bytes 2024-09-14T05:46:22.085 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 3180551168 bytes 2024-09-14T05:46:22.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 3241027584 bytes 2024-09-14T05:46:22.324 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 3261447168 bytes 2024-09-14T05:46:22.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 3295237120 bytes 2024-09-14T05:46:22.483 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-14T05:46:22.484 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-14T05:46:22.484 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2024-09-14T05:46:22.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-14T05:46:22.484+0000 7f7539804640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 40934) UID: 1000 2024-09-14T05:46:22.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 3344509952 bytes 2024-09-14T05:46:22.758 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 3453134848 bytes 2024-09-14T05:46:22.857 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 3470147584 bytes 2024-09-14T05:46:22.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7300 IOs, 3497243648 bytes 2024-09-14T05:46:23.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7400 IOs, 3518513152 bytes 2024-09-14T05:46:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7500 IOs, 3570164736 bytes 2024-09-14T05:46:23.299 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7600 IOs, 3609893888 bytes 2024-09-14T05:46:23.398 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7700 IOs, 3627991040 bytes 2024-09-14T05:46:23.500 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7800 IOs, 3652985856 bytes 2024-09-14T05:46:23.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7900 IOs, 3713170432 bytes 2024-09-14T05:46:23.745 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8000 IOs, 3738996736 bytes 2024-09-14T05:46:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8100 IOs, 3759592448 bytes 2024-09-14T05:46:23.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8200 IOs, 3766050816 bytes 2024-09-14T05:46:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8300 IOs, 3776509952 bytes 2024-09-14T05:46:24.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8400 IOs, 3805425664 bytes 2024-09-14T05:46:24.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8500 IOs, 3880062976 bytes 2024-09-14T05:46:24.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8600 IOs, 3959940096 bytes 2024-09-14T05:46:24.485 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-14T05:46:24.486 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T05:46:24.486 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-14T05:46:24.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-14T05:46:24.486 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-14T05:46:24.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-14T05:46:24.490 INFO:tasks.rbd_mirror_thrash:waiting for 115 secs before thrashing 2024-09-14T05:46:24.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8700 IOs, 3981082624 bytes 2024-09-14T05:46:24.688 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8800 IOs, 4042065920 bytes 2024-09-14T05:46:24.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8900 IOs, 4088066048 bytes 2024-09-14T05:46:24.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9000 IOs, 4138856448 bytes 2024-09-14T05:46:25.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9100 IOs, 4184722432 bytes 2024-09-14T05:46:25.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9200 IOs, 4203873280 bytes 2024-09-14T05:46:25.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9300 IOs, 4231178240 bytes 2024-09-14T05:46:25.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9400 IOs, 4277484544 bytes 2024-09-14T05:46:25.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9500 IOs, 4279678976 bytes 2024-09-14T05:46:25.674 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9600 IOs, 4383253504 bytes 2024-09-14T05:46:25.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:46:25.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:46:25.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:46:25.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:46:25.784 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:46:25.784 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_12 2024-09-14T05:46:25.784 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_12 snapshot 128M 2024-09-14T05:46:25.784 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:46:25.784 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:46:25.784 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:46:25.784 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:46:25.784 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_12 2024-09-14T05:46:25.784 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:46:25.784 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:46:25.784 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-14T05:46:25.784 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:46:25.785 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_12 128M 2024-09-14T05:46:25.785 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:46:25.785 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:46:25.785 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:46:25.785 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:46:25.785 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_12 2024-09-14T05:46:25.785 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:46:25.785 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T05:46:25.785 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-14T05:46:25.785 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-14T05:46:25.785 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:46:25.785 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_12 2024-09-14T05:46:25.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T05:46:25.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:46:25.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-14T05:46:25.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:46:25.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_12 snapshot 2024-09-14T05:46:25.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:46:25.835 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:46:25.835 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_12 2024-09-14T05:46:25.835 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:46:25.835 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_12 snapshot 2024-09-14T05:46:26.737 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T05:46:26.747 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_12 2024-09-14T05:46:26.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_12': 2024-09-14T05:46:26.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:46:26.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:46:26.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T05:46:26.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 12f02c4afee5 2024-09-14T05:46:26.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.12f02c4afee5 2024-09-14T05:46:26.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:46:26.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T05:46:26.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:46:26.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:46:26.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:46:25 2024 2024-09-14T05:46:26.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:46:25 2024 2024-09-14T05:46:26.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:46:25 2024 2024-09-14T05:46:26.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:46:26.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:46:26.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 7b610a90-6bb9-4ffb-8270-bbf747673d2a 2024-09-14T05:46:26.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T05:46:26.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:46:26.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_12 2024-09-14T05:46:26.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:46:26.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:46:26.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_12 2024-09-14T05:46:26.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:46:26.795 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=8 2024-09-14T05:46:26.795 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:46:26.795 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_12 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:46:27.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 59499520 bytes 2024-09-14T05:46:27.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 109470720 bytes 2024-09-14T05:46:27.229 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 149305344 bytes 2024-09-14T05:46:27.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 208146432 bytes 2024-09-14T05:46:27.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 219817984 bytes 2024-09-14T05:46:27.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 220430336 bytes 2024-09-14T05:46:27.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 228280320 bytes 2024-09-14T05:46:27.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 258791424 bytes 2024-09-14T05:46:27.787 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 268126208 bytes 2024-09-14T05:46:27.911 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 334697472 bytes 2024-09-14T05:46:28.018 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 357601280 bytes 2024-09-14T05:46:28.088 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 358204416 bytes 2024-09-14T05:46:28.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 365312000 bytes 2024-09-14T05:46:28.296 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 433732608 bytes 2024-09-14T05:46:28.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 484494336 bytes 2024-09-14T05:46:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 525749248 bytes 2024-09-14T05:46:28.639 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 553364480 bytes 2024-09-14T05:46:28.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 581172224 bytes 2024-09-14T05:46:28.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 625038336 bytes 2024-09-14T05:46:28.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 662697984 bytes 2024-09-14T05:46:29.054 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 664493056 bytes 2024-09-14T05:46:29.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 724764672 bytes 2024-09-14T05:46:29.295 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 770601984 bytes 2024-09-14T05:46:29.423 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 805531648 bytes 2024-09-14T05:46:29.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 820618240 bytes 2024-09-14T05:46:29.639 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 871326720 bytes 2024-09-14T05:46:29.801 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 964500480 bytes 2024-09-14T05:46:29.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1030301696 bytes 2024-09-14T05:46:30.056 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1080171520 bytes 2024-09-14T05:46:30.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1150917632 bytes 2024-09-14T05:46:30.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1199049728 bytes 2024-09-14T05:46:30.470 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1278396416 bytes 2024-09-14T05:46:30.548 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1284217856 bytes 2024-09-14T05:46:30.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1306599424 bytes 2024-09-14T05:46:30.773 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1354691584 bytes 2024-09-14T05:46:30.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1355291648 bytes 2024-09-14T05:46:30.930 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1365634048 bytes 2024-09-14T05:46:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1424538624 bytes 2024-09-14T05:46:31.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1442780160 bytes 2024-09-14T05:46:31.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1472062464 bytes 2024-09-14T05:46:31.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1497871360 bytes 2024-09-14T05:46:31.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1541177344 bytes 2024-09-14T05:46:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1597434880 bytes 2024-09-14T05:46:31.724 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1645673472 bytes 2024-09-14T05:46:31.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1692965888 bytes 2024-09-14T05:46:31.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1697350656 bytes 2024-09-14T05:46:32.073 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 1782107136 bytes 2024-09-14T05:46:32.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 1782722560 bytes 2024-09-14T05:46:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 1814990848 bytes 2024-09-14T05:46:32.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 1842870272 bytes 2024-09-14T05:46:32.437 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 1847222272 bytes 2024-09-14T05:46:32.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 1877658624 bytes 2024-09-14T05:46:32.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 1928623104 bytes 2024-09-14T05:46:32.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 1964208128 bytes 2024-09-14T05:46:32.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 1977675776 bytes 2024-09-14T05:46:33.035 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2086433792 bytes 2024-09-14T05:46:33.221 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2185526272 bytes 2024-09-14T05:46:33.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2251145216 bytes 2024-09-14T05:46:33.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2300243968 bytes 2024-09-14T05:46:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2370115584 bytes 2024-09-14T05:46:33.784 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2444399616 bytes 2024-09-14T05:46:33.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2520901632 bytes 2024-09-14T05:46:34.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2567417856 bytes 2024-09-14T05:46:34.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2622667776 bytes 2024-09-14T05:46:34.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2638006272 bytes 2024-09-14T05:46:34.425 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 2724666368 bytes 2024-09-14T05:46:34.546 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 2771517440 bytes 2024-09-14T05:46:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 2800561152 bytes 2024-09-14T05:46:34.776 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 2854452224 bytes 2024-09-14T05:46:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:46:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:46:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:46:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:46:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:46:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_13 2024-09-14T05:46:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_13 snapshot 128M 2024-09-14T05:46:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:46:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:46:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:46:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:46:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_13 2024-09-14T05:46:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:46:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:46:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-14T05:46:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:46:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_13 128M 2024-09-14T05:46:34.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:46:34.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:46:34.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:46:34.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:46:34.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_13 2024-09-14T05:46:34.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:46:34.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T05:46:34.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-14T05:46:34.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-14T05:46:34.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:46:34.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_13 2024-09-14T05:46:34.852 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T05:46:34.852 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:46:34.852 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-14T05:46:34.852 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:46:34.852 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_13 snapshot 2024-09-14T05:46:34.852 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:46:34.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:46:34.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_13 2024-09-14T05:46:34.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:46:34.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_13 snapshot 2024-09-14T05:46:35.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T05:46:35.773 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_13 2024-09-14T05:46:35.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_13': 2024-09-14T05:46:35.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:46:35.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:46:35.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T05:46:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 12fb4b2cb41d 2024-09-14T05:46:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.12fb4b2cb41d 2024-09-14T05:46:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:46:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T05:46:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:46:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:46:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:46:34 2024 2024-09-14T05:46:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:46:34 2024 2024-09-14T05:46:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:46:34 2024 2024-09-14T05:46:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:46:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:46:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 03d4b3e9-bc59-4da0-9ad0-74a331dde7bb 2024-09-14T05:46:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T05:46:35.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:46:35.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_13 2024-09-14T05:46:35.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:46:35.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:46:35.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_13 2024-09-14T05:46:35.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:46:35.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=7 2024-09-14T05:46:35.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:46:35.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_13 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:46:35.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 18188288 bytes 2024-09-14T05:46:36.102 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 66931712 bytes 2024-09-14T05:46:36.229 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 120031232 bytes 2024-09-14T05:46:36.325 INFO:tasks.ceph.cluster2.osd.0.smithi191.stderr:2024-09-14T05:46:36.323+0000 7f75dca16640 -1 reset not still connected to 0x55ea7c2e6dd0 2024-09-14T05:46:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 153140224 bytes 2024-09-14T05:46:36.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 196208640 bytes 2024-09-14T05:46:36.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 232501248 bytes 2024-09-14T05:46:36.658 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 247496704 bytes 2024-09-14T05:46:36.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 278204416 bytes 2024-09-14T05:46:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 351451136 bytes 2024-09-14T05:46:36.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 355361792 bytes 2024-09-14T05:46:37.076 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 380014592 bytes 2024-09-14T05:46:37.198 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 408196096 bytes 2024-09-14T05:46:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 518052864 bytes 2024-09-14T05:46:37.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 591458304 bytes 2024-09-14T05:46:37.625 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 610324480 bytes 2024-09-14T05:46:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 638524416 bytes 2024-09-14T05:46:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 726255616 bytes 2024-09-14T05:46:38.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 782270464 bytes 2024-09-14T05:46:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 808073216 bytes 2024-09-14T05:46:38.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 856052736 bytes 2024-09-14T05:46:38.407 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 922000384 bytes 2024-09-14T05:46:38.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 936505344 bytes 2024-09-14T05:46:38.637 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1000542208 bytes 2024-09-14T05:46:38.736 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1021824000 bytes 2024-09-14T05:46:38.848 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1054894080 bytes 2024-09-14T05:46:38.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1107391488 bytes 2024-09-14T05:46:39.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1144984576 bytes 2024-09-14T05:46:39.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1145611264 bytes 2024-09-14T05:46:39.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1195508736 bytes 2024-09-14T05:46:39.421 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1255116800 bytes 2024-09-14T05:46:39.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1279042560 bytes 2024-09-14T05:46:39.655 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1372319744 bytes 2024-09-14T05:46:39.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1424898048 bytes 2024-09-14T05:46:39.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1441620992 bytes 2024-09-14T05:46:40.014 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1461791744 bytes 2024-09-14T05:46:40.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1514884096 bytes 2024-09-14T05:46:40.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1595404288 bytes 2024-09-14T05:46:40.402 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1628181504 bytes 2024-09-14T05:46:40.595 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1754128384 bytes 2024-09-14T05:46:40.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1798093824 bytes 2024-09-14T05:46:40.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1821053952 bytes 2024-09-14T05:46:40.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1881562112 bytes 2024-09-14T05:46:41.035 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1882707968 bytes 2024-09-14T05:46:41.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1915165696 bytes 2024-09-14T05:46:41.262 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1958045696 bytes 2024-09-14T05:46:41.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 2021402624 bytes 2024-09-14T05:46:41.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 2098151424 bytes 2024-09-14T05:46:41.689 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2161875968 bytes 2024-09-14T05:46:41.800 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2197539840 bytes 2024-09-14T05:46:41.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2231793664 bytes 2024-09-14T05:46:41.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2232357888 bytes 2024-09-14T05:46:42.089 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2262791168 bytes 2024-09-14T05:46:42.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2273011712 bytes 2024-09-14T05:46:42.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2322790400 bytes 2024-09-14T05:46:42.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2372568064 bytes 2024-09-14T05:46:42.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2396932096 bytes 2024-09-14T05:46:42.832 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:46:42.832 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:46:42.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:46:42.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:46:42.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:46:42.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_14 2024-09-14T05:46:42.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_14 snapshot 128M 2024-09-14T05:46:42.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:46:42.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:46:42.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:46:42.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:46:42.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_14 2024-09-14T05:46:42.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:46:42.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:46:42.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-14T05:46:42.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:46:42.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_14 128M 2024-09-14T05:46:42.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:46:42.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:46:42.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:46:42.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:46:42.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_14 2024-09-14T05:46:42.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:46:42.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T05:46:42.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-14T05:46:42.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-14T05:46:42.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:46:42.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_14 2024-09-14T05:46:42.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T05:46:42.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:46:42.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-14T05:46:42.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:46:42.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_14 snapshot 2024-09-14T05:46:42.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:46:42.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:46:42.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_14 2024-09-14T05:46:42.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:46:42.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_14 snapshot 2024-09-14T05:46:43.776 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T05:46:43.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_14 2024-09-14T05:46:43.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_14': 2024-09-14T05:46:43.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:46:43.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:46:43.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T05:46:43.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 1307ba4dc232 2024-09-14T05:46:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.1307ba4dc232 2024-09-14T05:46:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:46:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T05:46:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:46:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:46:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:46:42 2024 2024-09-14T05:46:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:46:42 2024 2024-09-14T05:46:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:46:42 2024 2024-09-14T05:46:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:46:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:46:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 72651fba-3b42-454d-a354-6f5b7be1dd8d 2024-09-14T05:46:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T05:46:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:46:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_14 2024-09-14T05:46:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:46:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:46:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_14 2024-09-14T05:46:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:46:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=5 2024-09-14T05:46:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:46:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_14 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:46:44.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 36935680 bytes 2024-09-14T05:46:44.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 78076928 bytes 2024-09-14T05:46:44.251 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 125775872 bytes 2024-09-14T05:46:44.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 214471680 bytes 2024-09-14T05:46:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 239975424 bytes 2024-09-14T05:46:44.595 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 240596992 bytes 2024-09-14T05:46:44.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 262598656 bytes 2024-09-14T05:46:44.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 263188480 bytes 2024-09-14T05:46:44.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 268201984 bytes 2024-09-14T05:46:45.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 355126272 bytes 2024-09-14T05:46:45.118 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 355728384 bytes 2024-09-14T05:46:45.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 422095872 bytes 2024-09-14T05:46:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 456316928 bytes 2024-09-14T05:46:45.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 496922624 bytes 2024-09-14T05:46:45.588 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 504086528 bytes 2024-09-14T05:46:45.739 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 574212096 bytes 2024-09-14T05:46:45.836 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 605175808 bytes 2024-09-14T05:46:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 637923328 bytes 2024-09-14T05:46:46.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 733585408 bytes 2024-09-14T05:46:46.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 750824448 bytes 2024-09-14T05:46:46.392 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 819854336 bytes 2024-09-14T05:46:46.524 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 867051520 bytes 2024-09-14T05:46:46.610 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 880461824 bytes 2024-09-14T05:46:46.722 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 906382336 bytes 2024-09-14T05:46:46.857 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 963485696 bytes 2024-09-14T05:46:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 990851072 bytes 2024-09-14T05:46:47.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1080236032 bytes 2024-09-14T05:46:47.220 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1111690240 bytes 2024-09-14T05:46:47.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1125361664 bytes 2024-09-14T05:46:47.399 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1133686784 bytes 2024-09-14T05:46:47.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1170359296 bytes 2024-09-14T05:46:47.626 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1218580480 bytes 2024-09-14T05:46:47.751 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1253614592 bytes 2024-09-14T05:46:47.889 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1309534208 bytes 2024-09-14T05:46:48.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1347739648 bytes 2024-09-14T05:46:48.117 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1383196672 bytes 2024-09-14T05:46:48.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1427561472 bytes 2024-09-14T05:46:48.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1503416320 bytes 2024-09-14T05:46:48.489 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1524086784 bytes 2024-09-14T05:46:48.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1545684992 bytes 2024-09-14T05:46:48.751 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1632276480 bytes 2024-09-14T05:46:48.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:46:48.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:46:48.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:46:48.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:46:48.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:46:48.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_15 2024-09-14T05:46:48.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_15 snapshot 128M 2024-09-14T05:46:48.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:46:48.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:46:48.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:46:48.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:46:48.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_15 2024-09-14T05:46:48.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:46:48.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:46:48.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-14T05:46:48.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:46:48.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_15 128M 2024-09-14T05:46:48.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:46:48.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:46:48.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:46:48.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:46:48.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_15 2024-09-14T05:46:48.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:46:48.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T05:46:48.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-14T05:46:48.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-14T05:46:48.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:46:48.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_15 2024-09-14T05:46:48.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T05:46:48.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:46:48.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-14T05:46:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:46:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_15 snapshot 2024-09-14T05:46:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:46:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:46:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_15 2024-09-14T05:46:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:46:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_15 snapshot 2024-09-14T05:46:49.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T05:46:49.800 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_15 2024-09-14T05:46:49.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_15': 2024-09-14T05:46:49.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:46:49.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:46:49.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T05:46:49.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 13132735b8be 2024-09-14T05:46:49.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.13132735b8be 2024-09-14T05:46:49.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:46:49.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T05:46:49.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:46:49.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:46:49.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:46:48 2024 2024-09-14T05:46:49.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:46:48 2024 2024-09-14T05:46:49.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:46:48 2024 2024-09-14T05:46:49.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:46:49.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:46:49.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 8e5e2fa0-ccc2-4850-a80c-d8985387aa83 2024-09-14T05:46:49.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T05:46:49.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:46:49.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_15 2024-09-14T05:46:49.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:46:49.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:46:49.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_15 2024-09-14T05:46:49.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:46:49.848 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=9 2024-09-14T05:46:49.848 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:46:49.848 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_15 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:46:50.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 12324864 bytes 2024-09-14T05:46:50.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 45778944 bytes 2024-09-14T05:46:50.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 47085568 bytes 2024-09-14T05:46:50.325 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 81941504 bytes 2024-09-14T05:46:50.478 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 158085120 bytes 2024-09-14T05:46:50.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 161577984 bytes 2024-09-14T05:46:50.684 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 198419456 bytes 2024-09-14T05:46:50.800 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 231294976 bytes 2024-09-14T05:46:50.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 278782976 bytes 2024-09-14T05:46:51.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 325539840 bytes 2024-09-14T05:46:51.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 399165440 bytes 2024-09-14T05:46:51.302 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 426706944 bytes 2024-09-14T05:46:51.437 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 482156544 bytes 2024-09-14T05:46:51.538 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 509934592 bytes 2024-09-14T05:46:51.692 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 585486336 bytes 2024-09-14T05:46:51.836 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 642651136 bytes 2024-09-14T05:46:51.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 678417408 bytes 2024-09-14T05:46:52.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 712633344 bytes 2024-09-14T05:46:52.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 723876864 bytes 2024-09-14T05:46:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 776778752 bytes 2024-09-14T05:46:52.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 831901696 bytes 2024-09-14T05:46:52.515 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 857233408 bytes 2024-09-14T05:46:52.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 877337600 bytes 2024-09-14T05:46:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 887112704 bytes 2024-09-14T05:46:52.831 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 944262144 bytes 2024-09-14T05:46:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1105490944 bytes 2024-09-14T05:46:53.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1131068416 bytes 2024-09-14T05:46:53.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1219209216 bytes 2024-09-14T05:46:53.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1273915392 bytes 2024-09-14T05:46:53.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1325457408 bytes 2024-09-14T05:46:53.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1355540480 bytes 2024-09-14T05:46:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1402611712 bytes 2024-09-14T05:46:53.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1481336832 bytes 2024-09-14T05:46:54.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1580436480 bytes 2024-09-14T05:46:54.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1645058048 bytes 2024-09-14T05:46:54.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1704197120 bytes 2024-09-14T05:46:54.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1746355200 bytes 2024-09-14T05:46:54.681 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1767849984 bytes 2024-09-14T05:46:54.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1813741568 bytes 2024-09-14T05:46:54.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1849104384 bytes 2024-09-14T05:46:55.051 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1904447488 bytes 2024-09-14T05:46:55.185 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1964499968 bytes 2024-09-14T05:46:55.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1993652224 bytes 2024-09-14T05:46:55.440 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 2069076992 bytes 2024-09-14T05:46:55.549 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 2102491136 bytes 2024-09-14T05:46:55.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 2161141760 bytes 2024-09-14T05:46:55.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 2163000320 bytes 2024-09-14T05:46:55.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2206735360 bytes 2024-09-14T05:46:55.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2241460224 bytes 2024-09-14T05:46:56.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2267693056 bytes 2024-09-14T05:46:56.221 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2314426368 bytes 2024-09-14T05:46:56.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2427735040 bytes 2024-09-14T05:46:56.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2437123072 bytes 2024-09-14T05:46:56.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2494132224 bytes 2024-09-14T05:46:56.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2537552896 bytes 2024-09-14T05:46:56.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2538168320 bytes 2024-09-14T05:46:56.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2580328448 bytes 2024-09-14T05:46:57.084 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2630750208 bytes 2024-09-14T05:46:57.187 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2660616192 bytes 2024-09-14T05:46:57.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2760600576 bytes 2024-09-14T05:46:57.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2819879936 bytes 2024-09-14T05:46:57.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2823642112 bytes 2024-09-14T05:46:57.708 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2865760256 bytes 2024-09-14T05:46:57.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2884814848 bytes 2024-09-14T05:46:57.931 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2931560448 bytes 2024-09-14T05:46:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 3011357696 bytes 2024-09-14T05:46:58.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 3041655808 bytes 2024-09-14T05:46:58.287 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 3052221440 bytes 2024-09-14T05:46:58.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 3071175680 bytes 2024-09-14T05:46:58.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 3142189056 bytes 2024-09-14T05:46:58.623 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 3170478080 bytes 2024-09-14T05:46:58.844 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 3322138624 bytes 2024-09-14T05:46:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:46:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:46:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:46:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:46:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:46:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_16 2024-09-14T05:46:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_16 snapshot 128M 2024-09-14T05:46:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:46:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:46:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:46:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:46:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_16 2024-09-14T05:46:58.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:46:58.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:46:58.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-14T05:46:58.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:46:58.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_16 128M 2024-09-14T05:46:58.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:46:58.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:46:58.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:46:58.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:46:58.863 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_16 2024-09-14T05:46:58.863 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:46:58.863 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T05:46:58.863 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-14T05:46:58.863 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-14T05:46:58.863 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:46:58.863 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_16 2024-09-14T05:46:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T05:46:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:46:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-14T05:46:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:46:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_16 snapshot 2024-09-14T05:46:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:46:58.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:46:58.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_16 2024-09-14T05:46:58.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:46:58.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_16 snapshot 2024-09-14T05:46:59.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T05:46:59.815 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_16 2024-09-14T05:46:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_16': 2024-09-14T05:46:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:46:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:46:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T05:46:59.855 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 131fb95c433d 2024-09-14T05:46:59.855 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.131fb95c433d 2024-09-14T05:46:59.855 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:46:59.855 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T05:46:59.855 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:46:59.855 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:46:59.855 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:46:58 2024 2024-09-14T05:46:59.855 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:46:58 2024 2024-09-14T05:46:59.855 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:46:58 2024 2024-09-14T05:46:59.855 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:46:59.855 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:46:59.855 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 1fcc0d0b-34ef-46ef-b563-beb55715aefe 2024-09-14T05:46:59.855 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T05:46:59.858 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:46:59.858 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_16 2024-09-14T05:46:59.858 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:46:59.858 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:46:59.858 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_16 2024-09-14T05:46:59.858 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:46:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=6 2024-09-14T05:46:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:46:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_16 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:47:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 47558656 bytes 2024-09-14T05:47:00.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 113150976 bytes 2024-09-14T05:47:00.333 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 173578240 bytes 2024-09-14T05:47:00.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 174184448 bytes 2024-09-14T05:47:00.489 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 181824512 bytes 2024-09-14T05:47:00.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 219837440 bytes 2024-09-14T05:47:00.721 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 256539648 bytes 2024-09-14T05:47:00.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 265714688 bytes 2024-09-14T05:47:00.926 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 311153664 bytes 2024-09-14T05:47:01.012 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 323223552 bytes 2024-09-14T05:47:01.160 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 391771136 bytes 2024-09-14T05:47:01.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 444064768 bytes 2024-09-14T05:47:01.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 479376384 bytes 2024-09-14T05:47:01.515 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 529684480 bytes 2024-09-14T05:47:01.654 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 590126080 bytes 2024-09-14T05:47:01.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 641411072 bytes 2024-09-14T05:47:01.925 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 694047744 bytes 2024-09-14T05:47:02.062 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 757617664 bytes 2024-09-14T05:47:02.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 782268416 bytes 2024-09-14T05:47:02.373 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 904869888 bytes 2024-09-14T05:47:02.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 965029888 bytes 2024-09-14T05:47:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 1009616896 bytes 2024-09-14T05:47:02.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1025800192 bytes 2024-09-14T05:47:02.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1088826368 bytes 2024-09-14T05:47:02.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1126806528 bytes 2024-09-14T05:47:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1177255936 bytes 2024-09-14T05:47:03.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1248155648 bytes 2024-09-14T05:47:03.406 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1348620288 bytes 2024-09-14T05:47:03.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1385280512 bytes 2024-09-14T05:47:03.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1443485696 bytes 2024-09-14T05:47:03.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1515501568 bytes 2024-09-14T05:47:03.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1582140416 bytes 2024-09-14T05:47:04.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1591901184 bytes 2024-09-14T05:47:04.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1661615104 bytes 2024-09-14T05:47:04.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1729497088 bytes 2024-09-14T05:47:04.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1749006336 bytes 2024-09-14T05:47:04.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1776057344 bytes 2024-09-14T05:47:04.600 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1796884480 bytes 2024-09-14T05:47:04.719 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1824145408 bytes 2024-09-14T05:47:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1852576768 bytes 2024-09-14T05:47:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1889970176 bytes 2024-09-14T05:47:05.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1928045568 bytes 2024-09-14T05:47:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1938613248 bytes 2024-09-14T05:47:05.262 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1988985856 bytes 2024-09-14T05:47:05.423 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 2079702016 bytes 2024-09-14T05:47:05.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 2175373312 bytes 2024-09-14T05:47:05.720 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 2213120000 bytes 2024-09-14T05:47:05.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:47:05.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:47:05.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:47:05.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:47:05.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:47:05.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_17 2024-09-14T05:47:05.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_17 snapshot 128M 2024-09-14T05:47:05.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:47:05.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:47:05.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:47:05.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:47:05.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_17 2024-09-14T05:47:05.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:47:05.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:47:05.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-14T05:47:05.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:47:05.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_17 128M 2024-09-14T05:47:05.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:47:05.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:47:05.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:47:05.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:47:05.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_17 2024-09-14T05:47:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:47:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T05:47:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-14T05:47:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-14T05:47:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:47:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_17 2024-09-14T05:47:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T05:47:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:47:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-14T05:47:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:47:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_17 snapshot 2024-09-14T05:47:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:47:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:47:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_17 2024-09-14T05:47:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:47:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_17 snapshot 2024-09-14T05:47:06.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T05:47:06.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_17 2024-09-14T05:47:06.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_17': 2024-09-14T05:47:06.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:47:06.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:47:06.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T05:47:06.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 132bcd40d375 2024-09-14T05:47:06.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.132bcd40d375 2024-09-14T05:47:06.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:47:06.863 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T05:47:06.863 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:47:06.863 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:47:06.863 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:47:05 2024 2024-09-14T05:47:06.863 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:47:05 2024 2024-09-14T05:47:06.863 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:47:05 2024 2024-09-14T05:47:06.863 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:47:06.863 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:47:06.863 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 5c00a5fc-12ce-46ba-b4c0-0e6aa090fe1a 2024-09-14T05:47:06.863 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T05:47:06.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:47:06.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_17 2024-09-14T05:47:06.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:47:06.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:47:06.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_17 2024-09-14T05:47:06.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:47:06.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=11 2024-09-14T05:47:06.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:47:06.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_17 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:47:07.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 38218752 bytes 2024-09-14T05:47:07.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 105180160 bytes 2024-09-14T05:47:07.328 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 133523456 bytes 2024-09-14T05:47:07.426 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 150706176 bytes 2024-09-14T05:47:07.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 233284608 bytes 2024-09-14T05:47:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 300603392 bytes 2024-09-14T05:47:07.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 341475328 bytes 2024-09-14T05:47:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 402470912 bytes 2024-09-14T05:47:08.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 443197440 bytes 2024-09-14T05:47:08.270 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 501682176 bytes 2024-09-14T05:47:08.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 516932608 bytes 2024-09-14T05:47:08.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 580690944 bytes 2024-09-14T05:47:08.690 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 673139712 bytes 2024-09-14T05:47:08.800 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 698505216 bytes 2024-09-14T05:47:08.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 749914112 bytes 2024-09-14T05:47:09.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 773995520 bytes 2024-09-14T05:47:09.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 816376832 bytes 2024-09-14T05:47:09.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 858531840 bytes 2024-09-14T05:47:09.403 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 899087360 bytes 2024-09-14T05:47:09.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 913912832 bytes 2024-09-14T05:47:09.653 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 992794624 bytes 2024-09-14T05:47:09.749 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 1020408832 bytes 2024-09-14T05:47:09.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1070105600 bytes 2024-09-14T05:47:10.020 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1125230592 bytes 2024-09-14T05:47:10.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1167867904 bytes 2024-09-14T05:47:10.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1217777664 bytes 2024-09-14T05:47:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1271577600 bytes 2024-09-14T05:47:10.556 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1335324672 bytes 2024-09-14T05:47:10.645 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1343109120 bytes 2024-09-14T05:47:10.748 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1369523200 bytes 2024-09-14T05:47:10.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1396728832 bytes 2024-09-14T05:47:10.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1437080576 bytes 2024-09-14T05:47:11.068 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1448950784 bytes 2024-09-14T05:47:11.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1472362496 bytes 2024-09-14T05:47:11.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1532014592 bytes 2024-09-14T05:47:11.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1573846016 bytes 2024-09-14T05:47:11.571 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1622758400 bytes 2024-09-14T05:47:11.704 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1681540096 bytes 2024-09-14T05:47:11.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1700096000 bytes 2024-09-14T05:47:11.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1729912832 bytes 2024-09-14T05:47:12.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1765742592 bytes 2024-09-14T05:47:12.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1806068736 bytes 2024-09-14T05:47:12.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1854437376 bytes 2024-09-14T05:47:12.423 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1929912320 bytes 2024-09-14T05:47:12.545 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1963888640 bytes 2024-09-14T05:47:12.654 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1991248896 bytes 2024-09-14T05:47:12.787 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 2037827584 bytes 2024-09-14T05:47:12.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2082069504 bytes 2024-09-14T05:47:13.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2129697792 bytes 2024-09-14T05:47:13.228 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2161622016 bytes 2024-09-14T05:47:13.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2191465472 bytes 2024-09-14T05:47:13.499 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2267662336 bytes 2024-09-14T05:47:13.645 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2320385024 bytes 2024-09-14T05:47:13.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2323913728 bytes 2024-09-14T05:47:13.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2350538752 bytes 2024-09-14T05:47:13.951 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2384791552 bytes 2024-09-14T05:47:14.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2395936768 bytes 2024-09-14T05:47:14.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2397081600 bytes 2024-09-14T05:47:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2412300288 bytes 2024-09-14T05:47:14.383 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2484504576 bytes 2024-09-14T05:47:14.552 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2570624000 bytes 2024-09-14T05:47:14.667 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2602949632 bytes 2024-09-14T05:47:14.816 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2674361344 bytes 2024-09-14T05:47:14.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2722907136 bytes 2024-09-14T05:47:15.076 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2765905920 bytes 2024-09-14T05:47:15.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 2798282752 bytes 2024-09-14T05:47:15.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 2802132992 bytes 2024-09-14T05:47:15.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 2840099840 bytes 2024-09-14T05:47:15.589 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 2953058304 bytes 2024-09-14T05:47:15.682 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 2957279232 bytes 2024-09-14T05:47:15.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 3006179328 bytes 2024-09-14T05:47:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 3076502528 bytes 2024-09-14T05:47:16.074 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7300 IOs, 3103431680 bytes 2024-09-14T05:47:16.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7400 IOs, 3104046080 bytes 2024-09-14T05:47:16.294 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7500 IOs, 3154239488 bytes 2024-09-14T05:47:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7600 IOs, 3199087616 bytes 2024-09-14T05:47:16.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7700 IOs, 3220943872 bytes 2024-09-14T05:47:16.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7800 IOs, 3253281792 bytes 2024-09-14T05:47:16.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7900 IOs, 3257617408 bytes 2024-09-14T05:47:16.811 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8000 IOs, 3270307840 bytes 2024-09-14T05:47:16.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8100 IOs, 3320974336 bytes 2024-09-14T05:47:17.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8200 IOs, 3347388416 bytes 2024-09-14T05:47:17.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8300 IOs, 3391720448 bytes 2024-09-14T05:47:17.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8400 IOs, 3414877184 bytes 2024-09-14T05:47:17.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8500 IOs, 3448216576 bytes 2024-09-14T05:47:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8600 IOs, 3576349696 bytes 2024-09-14T05:47:17.678 INFO:tasks.ceph.cluster2.osd.1.smithi191.stderr:2024-09-14T05:47:17.677+0000 7f9375a93640 -1 reset not still connected to 0x5611e9425790 2024-09-14T05:47:17.745 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8700 IOs, 3617683456 bytes 2024-09-14T05:47:17.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:47:17.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:47:17.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:47:17.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:47:17.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:47:17.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_18 2024-09-14T05:47:17.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_18 snapshot 128M 2024-09-14T05:47:17.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:47:17.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:47:17.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:47:17.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:47:17.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_18 2024-09-14T05:47:17.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:47:17.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:47:17.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-14T05:47:17.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:47:17.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_18 128M 2024-09-14T05:47:17.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:47:17.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:47:17.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:47:17.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:47:17.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_18 2024-09-14T05:47:17.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:47:17.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T05:47:17.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-14T05:47:17.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-14T05:47:17.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:47:17.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_18 2024-09-14T05:47:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T05:47:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:47:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-14T05:47:17.932 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:47:17.932 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_18 snapshot 2024-09-14T05:47:17.932 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:47:17.932 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:47:17.932 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_18 2024-09-14T05:47:17.932 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:47:17.932 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_18 snapshot 2024-09-14T05:47:18.851 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T05:47:18.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_18 2024-09-14T05:47:18.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_18': 2024-09-14T05:47:18.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:47:18.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:47:18.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T05:47:18.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 133781e6dbc8 2024-09-14T05:47:18.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.133781e6dbc8 2024-09-14T05:47:18.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:47:18.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T05:47:18.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:47:18.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:47:18.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:47:17 2024 2024-09-14T05:47:18.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:47:17 2024 2024-09-14T05:47:18.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:47:17 2024 2024-09-14T05:47:18.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:47:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:47:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 3d117385-98ae-4587-b863-f50b36e012f3 2024-09-14T05:47:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T05:47:18.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:47:18.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_18 2024-09-14T05:47:18.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:47:18.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:47:18.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_18 2024-09-14T05:47:18.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:47:18.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=12 2024-09-14T05:47:18.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:47:18.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_18 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:47:19.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 18085888 bytes 2024-09-14T05:47:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 40006656 bytes 2024-09-14T05:47:19.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 125142016 bytes 2024-09-14T05:47:19.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 134701056 bytes 2024-09-14T05:47:19.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 160533504 bytes 2024-09-14T05:47:19.636 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 175206400 bytes 2024-09-14T05:47:19.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 260396032 bytes 2024-09-14T05:47:19.911 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 298106880 bytes 2024-09-14T05:47:20.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 322282496 bytes 2024-09-14T05:47:20.185 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 421399552 bytes 2024-09-14T05:47:20.326 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 479214592 bytes 2024-09-14T05:47:20.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 519883776 bytes 2024-09-14T05:47:20.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 602067968 bytes 2024-09-14T05:47:20.689 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 602712064 bytes 2024-09-14T05:47:20.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 650236928 bytes 2024-09-14T05:47:20.926 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 682767360 bytes 2024-09-14T05:47:21.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 741033984 bytes 2024-09-14T05:47:21.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 767919104 bytes 2024-09-14T05:47:21.293 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 809700352 bytes 2024-09-14T05:47:21.400 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 842624000 bytes 2024-09-14T05:47:21.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 877605888 bytes 2024-09-14T05:47:21.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 910212096 bytes 2024-09-14T05:47:21.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 956244992 bytes 2024-09-14T05:47:21.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 981703680 bytes 2024-09-14T05:47:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 997664768 bytes 2024-09-14T05:47:22.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1045330944 bytes 2024-09-14T05:47:22.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1077912576 bytes 2024-09-14T05:47:22.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1111035904 bytes 2024-09-14T05:47:22.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1119390720 bytes 2024-09-14T05:47:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1130499072 bytes 2024-09-14T05:47:22.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1144180736 bytes 2024-09-14T05:47:22.708 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1195609088 bytes 2024-09-14T05:47:22.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1220874240 bytes 2024-09-14T05:47:22.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1255826432 bytes 2024-09-14T05:47:23.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1324104704 bytes 2024-09-14T05:47:23.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1340749824 bytes 2024-09-14T05:47:23.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1377444864 bytes 2024-09-14T05:47:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1485217792 bytes 2024-09-14T05:47:23.599 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1510959104 bytes 2024-09-14T05:47:23.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1595006976 bytes 2024-09-14T05:47:23.911 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1656732672 bytes 2024-09-14T05:47:24.021 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1687909376 bytes 2024-09-14T05:47:24.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1717352448 bytes 2024-09-14T05:47:24.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1717972992 bytes 2024-09-14T05:47:24.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1725529088 bytes 2024-09-14T05:47:24.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1726136320 bytes 2024-09-14T05:47:24.468 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 1726756864 bytes 2024-09-14T05:47:24.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 1765922816 bytes 2024-09-14T05:47:24.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 1848275968 bytes 2024-09-14T05:47:24.836 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 1848865792 bytes 2024-09-14T05:47:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 1909312512 bytes 2024-09-14T05:47:25.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 1936231424 bytes 2024-09-14T05:47:25.226 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2011037696 bytes 2024-09-14T05:47:25.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2050875392 bytes 2024-09-14T05:47:25.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2066851840 bytes 2024-09-14T05:47:25.564 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2106328064 bytes 2024-09-14T05:47:25.692 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2161274880 bytes 2024-09-14T05:47:25.859 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2258065408 bytes 2024-09-14T05:47:25.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2287566848 bytes 2024-09-14T05:47:26.113 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2337240064 bytes 2024-09-14T05:47:26.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2375199744 bytes 2024-09-14T05:47:26.333 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2405665792 bytes 2024-09-14T05:47:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2514409472 bytes 2024-09-14T05:47:26.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2648385536 bytes 2024-09-14T05:47:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2676574208 bytes 2024-09-14T05:47:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 2712819712 bytes 2024-09-14T05:47:27.088 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 2777120768 bytes 2024-09-14T05:47:27.190 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 2802155520 bytes 2024-09-14T05:47:27.291 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 2827028480 bytes 2024-09-14T05:47:27.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 2838972416 bytes 2024-09-14T05:47:27.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 2903799808 bytes 2024-09-14T05:47:27.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 2926417920 bytes 2024-09-14T05:47:27.771 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7300 IOs, 2993828864 bytes 2024-09-14T05:47:27.856 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7400 IOs, 3006397440 bytes 2024-09-14T05:47:27.991 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7500 IOs, 3051413504 bytes 2024-09-14T05:47:28.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7600 IOs, 3091426304 bytes 2024-09-14T05:47:28.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7700 IOs, 3131455488 bytes 2024-09-14T05:47:28.327 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7800 IOs, 3156113408 bytes 2024-09-14T05:47:28.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7900 IOs, 3191654400 bytes 2024-09-14T05:47:28.546 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8000 IOs, 3222715392 bytes 2024-09-14T05:47:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8100 IOs, 3349722112 bytes 2024-09-14T05:47:28.855 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8200 IOs, 3380042752 bytes 2024-09-14T05:47:28.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8300 IOs, 3410927616 bytes 2024-09-14T05:47:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8400 IOs, 3432446976 bytes 2024-09-14T05:47:29.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8500 IOs, 3494489088 bytes 2024-09-14T05:47:29.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8600 IOs, 3518850048 bytes 2024-09-14T05:47:29.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8700 IOs, 3564441600 bytes 2024-09-14T05:47:29.549 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8800 IOs, 3618372608 bytes 2024-09-14T05:47:29.668 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8900 IOs, 3646880768 bytes 2024-09-14T05:47:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9000 IOs, 3692887040 bytes 2024-09-14T05:47:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9100 IOs, 3715986432 bytes 2024-09-14T05:47:29.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9200 IOs, 3718984704 bytes 2024-09-14T05:47:30.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9300 IOs, 3780058112 bytes 2024-09-14T05:47:30.272 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9400 IOs, 3849680896 bytes 2024-09-14T05:47:30.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9500 IOs, 3891636224 bytes 2024-09-14T05:47:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9600 IOs, 3896814592 bytes 2024-09-14T05:47:30.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9700 IOs, 3950750720 bytes 2024-09-14T05:47:30.748 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9800 IOs, 3998434304 bytes 2024-09-14T05:47:30.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9900 IOs, 4061594624 bytes 2024-09-14T05:47:30.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:47:30.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:47:30.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:47:30.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:47:30.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:47:30.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_19 2024-09-14T05:47:30.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_19 snapshot 128M 2024-09-14T05:47:30.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:47:30.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:47:30.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:47:30.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:47:30.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_19 2024-09-14T05:47:30.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:47:30.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:47:30.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-14T05:47:30.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:47:30.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_19 128M 2024-09-14T05:47:30.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:47:30.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:47:30.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:47:30.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:47:30.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_19 2024-09-14T05:47:30.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:47:30.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T05:47:30.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-14T05:47:30.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-14T05:47:30.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:47:30.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_19 2024-09-14T05:47:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T05:47:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:47:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-14T05:47:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:47:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_19 snapshot 2024-09-14T05:47:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:47:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:47:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_19 2024-09-14T05:47:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:47:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_19 snapshot 2024-09-14T05:47:31.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T05:47:31.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_19 2024-09-14T05:47:31.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_19': 2024-09-14T05:47:31.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:47:31.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:47:31.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T05:47:31.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 1343633ec425 2024-09-14T05:47:31.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.1343633ec425 2024-09-14T05:47:31.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:47:31.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T05:47:31.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:47:31.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:47:31.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:47:30 2024 2024-09-14T05:47:31.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:47:30 2024 2024-09-14T05:47:31.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:47:30 2024 2024-09-14T05:47:31.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:47:31.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:47:31.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: a166af45-2737-4c1c-912c-26014f069f63 2024-09-14T05:47:31.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T05:47:31.925 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:47:31.925 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_19 2024-09-14T05:47:31.925 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:47:31.925 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:47:31.925 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_19 2024-09-14T05:47:31.925 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:47:31.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=6 2024-09-14T05:47:31.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:47:31.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_19 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:47:32.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 43934720 bytes 2024-09-14T05:47:32.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 68904960 bytes 2024-09-14T05:47:32.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 89152512 bytes 2024-09-14T05:47:32.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 101454848 bytes 2024-09-14T05:47:32.463 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6514433079269005) > weight (0.25) 2024-09-14T05:47:32.463 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7855173769413277) > weight (0.25) 2024-09-14T05:47:32.463 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.5754479396231627) > weight (0.25) 2024-09-14T05:47:32.463 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2024-09-14T05:47:32.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2024-09-14T05:47:32.464 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-09-14T05:47:32.465 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-14T05:47:32.464+0000 7fd6078ce640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 42190) UID: 1000 2024-09-14T05:47:32.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 119152640 bytes 2024-09-14T05:47:32.613 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 119762944 bytes 2024-09-14T05:47:32.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 128163840 bytes 2024-09-14T05:47:32.852 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 206453760 bytes 2024-09-14T05:47:32.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 242788352 bytes 2024-09-14T05:47:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 278443008 bytes 2024-09-14T05:47:33.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 328655872 bytes 2024-09-14T05:47:33.325 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 358719488 bytes 2024-09-14T05:47:33.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 395744256 bytes 2024-09-14T05:47:33.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 404553728 bytes 2024-09-14T05:47:33.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 420916224 bytes 2024-09-14T05:47:33.734 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 441052160 bytes 2024-09-14T05:47:33.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 485603328 bytes 2024-09-14T05:47:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 494982144 bytes 2024-09-14T05:47:34.096 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 588128256 bytes 2024-09-14T05:47:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 668983296 bytes 2024-09-14T05:47:34.423 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 713553920 bytes 2024-09-14T05:47:34.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 753669120 bytes 2024-09-14T05:47:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 782535680 bytes 2024-09-14T05:47:34.734 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 808776704 bytes 2024-09-14T05:47:34.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 888365056 bytes 2024-09-14T05:47:35.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 920941568 bytes 2024-09-14T05:47:35.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 964156416 bytes 2024-09-14T05:47:35.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 978893824 bytes 2024-09-14T05:47:35.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 999775232 bytes 2024-09-14T05:47:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1019674624 bytes 2024-09-14T05:47:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1048426496 bytes 2024-09-14T05:47:35.676 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1072783360 bytes 2024-09-14T05:47:35.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1118518272 bytes 2024-09-14T05:47:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1133164544 bytes 2024-09-14T05:47:36.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1169498112 bytes 2024-09-14T05:47:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1246067712 bytes 2024-09-14T05:47:36.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1321670656 bytes 2024-09-14T05:47:36.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1387473920 bytes 2024-09-14T05:47:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1427849216 bytes 2024-09-14T05:47:36.748 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1482286080 bytes 2024-09-14T05:47:36.899 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1561452544 bytes 2024-09-14T05:47:37.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1643019264 bytes 2024-09-14T05:47:37.198 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1684399104 bytes 2024-09-14T05:47:37.326 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1737378816 bytes 2024-09-14T05:47:37.465 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2024-09-14T05:47:37.466 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T05:47:37.466 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2024-09-14T05:47:37.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-09-14T05:47:37.467 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-09-14T05:47:37.468 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1812211712 bytes 2024-09-14T05:47:37.470 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-09-14T05:47:37.470 INFO:tasks.rbd_mirror_thrash:waiting for 109 secs before thrashing 2024-09-14T05:47:37.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1876139008 bytes 2024-09-14T05:47:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 1912605696 bytes 2024-09-14T05:47:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 1913725952 bytes 2024-09-14T05:47:37.932 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:47:37.932 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:47:37.932 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:47:37.932 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:47:37.932 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:47:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_20 2024-09-14T05:47:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_20 snapshot 128M 2024-09-14T05:47:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:47:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:47:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:47:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:47:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_20 2024-09-14T05:47:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:47:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:47:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-14T05:47:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:47:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_20 128M 2024-09-14T05:47:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:47:37.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:47:37.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:47:37.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:47:37.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_20 2024-09-14T05:47:37.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:47:37.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T05:47:37.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-14T05:47:37.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-14T05:47:37.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:47:37.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_20 2024-09-14T05:47:37.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T05:47:37.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:47:37.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-14T05:47:37.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:47:37.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_20 snapshot 2024-09-14T05:47:37.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:47:37.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:47:37.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_20 2024-09-14T05:47:37.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:47:37.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_20 snapshot 2024-09-14T05:47:38.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T05:47:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_20 2024-09-14T05:47:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_20': 2024-09-14T05:47:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:47:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:47:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T05:47:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 1355e47b6d70 2024-09-14T05:47:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.1355e47b6d70 2024-09-14T05:47:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:47:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T05:47:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:47:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:47:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:47:37 2024 2024-09-14T05:47:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:47:37 2024 2024-09-14T05:47:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:47:37 2024 2024-09-14T05:47:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:47:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:47:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 6654bddc-72e9-47b0-8061-615cff98334f 2024-09-14T05:47:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T05:47:38.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:47:38.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_20 2024-09-14T05:47:38.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:47:38.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:47:38.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_20 2024-09-14T05:47:38.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:47:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=7 2024-09-14T05:47:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:47:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_20 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:47:39.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 46643200 bytes 2024-09-14T05:47:39.286 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 105016320 bytes 2024-09-14T05:47:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 126073856 bytes 2024-09-14T05:47:39.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 198834176 bytes 2024-09-14T05:47:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 223254528 bytes 2024-09-14T05:47:39.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 294496256 bytes 2024-09-14T05:47:39.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 364453888 bytes 2024-09-14T05:47:40.056 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 414875648 bytes 2024-09-14T05:47:40.185 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 469384192 bytes 2024-09-14T05:47:40.293 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 500260864 bytes 2024-09-14T05:47:40.381 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 508850176 bytes 2024-09-14T05:47:40.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 530707456 bytes 2024-09-14T05:47:40.652 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 628775936 bytes 2024-09-14T05:47:40.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 679445504 bytes 2024-09-14T05:47:40.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 734014464 bytes 2024-09-14T05:47:41.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 819608576 bytes 2024-09-14T05:47:41.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 854952960 bytes 2024-09-14T05:47:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 929939456 bytes 2024-09-14T05:47:41.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 982636544 bytes 2024-09-14T05:47:41.552 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 983264256 bytes 2024-09-14T05:47:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 1062130688 bytes 2024-09-14T05:47:41.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 1078298624 bytes 2024-09-14T05:47:41.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1168342016 bytes 2024-09-14T05:47:42.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1190574080 bytes 2024-09-14T05:47:42.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1232381952 bytes 2024-09-14T05:47:42.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1306022912 bytes 2024-09-14T05:47:42.470 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1350481920 bytes 2024-09-14T05:47:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1367072768 bytes 2024-09-14T05:47:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1420878848 bytes 2024-09-14T05:47:42.795 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1443034112 bytes 2024-09-14T05:47:43.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1544458240 bytes 2024-09-14T05:47:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1569425408 bytes 2024-09-14T05:47:43.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1583938560 bytes 2024-09-14T05:47:43.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1640154112 bytes 2024-09-14T05:47:43.545 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1793765376 bytes 2024-09-14T05:47:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1871784960 bytes 2024-09-14T05:47:43.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1913275392 bytes 2024-09-14T05:47:43.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1964024832 bytes 2024-09-14T05:47:44.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1973257216 bytes 2024-09-14T05:47:44.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 2043974656 bytes 2024-09-14T05:47:44.291 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 2045990912 bytes 2024-09-14T05:47:44.393 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 2069897216 bytes 2024-09-14T05:47:44.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 2100570112 bytes 2024-09-14T05:47:44.618 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 2129892352 bytes 2024-09-14T05:47:44.731 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 2168570880 bytes 2024-09-14T05:47:44.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 2172751872 bytes 2024-09-14T05:47:44.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 2275361792 bytes 2024-09-14T05:47:45.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2281398272 bytes 2024-09-14T05:47:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2308185088 bytes 2024-09-14T05:47:45.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2362153984 bytes 2024-09-14T05:47:45.441 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2397378560 bytes 2024-09-14T05:47:45.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2408510464 bytes 2024-09-14T05:47:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2438458368 bytes 2024-09-14T05:47:45.781 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2492571648 bytes 2024-09-14T05:47:45.890 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2524838912 bytes 2024-09-14T05:47:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:47:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:47:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:47:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:47:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:47:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_21 2024-09-14T05:47:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_21 snapshot 128M 2024-09-14T05:47:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:47:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:47:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:47:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:47:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_21 2024-09-14T05:47:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:47:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:47:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-14T05:47:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:47:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_21 128M 2024-09-14T05:47:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:47:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:47:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:47:45.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:47:45.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_21 2024-09-14T05:47:45.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:47:45.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T05:47:45.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-14T05:47:45.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-14T05:47:45.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:47:45.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_21 2024-09-14T05:47:46.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T05:47:46.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:47:46.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-14T05:47:46.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:47:46.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_21 snapshot 2024-09-14T05:47:46.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:47:46.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:47:46.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_21 2024-09-14T05:47:46.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:47:46.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_21 snapshot 2024-09-14T05:47:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T05:47:46.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_21 2024-09-14T05:47:46.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_21': 2024-09-14T05:47:46.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:47:46.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:47:46.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T05:47:46.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 136188a59a64 2024-09-14T05:47:46.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.136188a59a64 2024-09-14T05:47:46.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:47:46.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T05:47:46.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:47:46.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:47:46.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:47:45 2024 2024-09-14T05:47:46.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:47:45 2024 2024-09-14T05:47:46.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:47:45 2024 2024-09-14T05:47:46.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:47:46.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:47:46.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: a03aba9f-8291-447c-bf72-b41385e7c989 2024-09-14T05:47:46.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T05:47:46.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:47:46.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_21 2024-09-14T05:47:46.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:47:46.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:47:46.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_21 2024-09-14T05:47:46.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:47:46.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=9 2024-09-14T05:47:46.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:47:46.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_21 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:47:47.187 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 60261376 bytes 2024-09-14T05:47:47.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 136623104 bytes 2024-09-14T05:47:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 159391744 bytes 2024-09-14T05:47:47.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 191315968 bytes 2024-09-14T05:47:47.673 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 231664640 bytes 2024-09-14T05:47:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 288616448 bytes 2024-09-14T05:47:47.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 289239040 bytes 2024-09-14T05:47:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 326189056 bytes 2024-09-14T05:47:48.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 354731008 bytes 2024-09-14T05:47:48.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 407431168 bytes 2024-09-14T05:47:48.440 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 488201216 bytes 2024-09-14T05:47:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 546642944 bytes 2024-09-14T05:47:48.703 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 596951040 bytes 2024-09-14T05:47:48.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 598512640 bytes 2024-09-14T05:47:48.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 677752832 bytes 2024-09-14T05:47:49.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 717471744 bytes 2024-09-14T05:47:49.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 808931328 bytes 2024-09-14T05:47:49.392 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 809527296 bytes 2024-09-14T05:47:49.499 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 837831680 bytes 2024-09-14T05:47:49.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 932844544 bytes 2024-09-14T05:47:49.836 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 1016950784 bytes 2024-09-14T05:47:49.926 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 1034574848 bytes 2024-09-14T05:47:50.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1104288768 bytes 2024-09-14T05:47:50.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1122002944 bytes 2024-09-14T05:47:50.331 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1203284992 bytes 2024-09-14T05:47:50.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1244971008 bytes 2024-09-14T05:47:50.557 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1263728640 bytes 2024-09-14T05:47:50.658 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1284686848 bytes 2024-09-14T05:47:50.785 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1328955392 bytes 2024-09-14T05:47:50.925 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1391081472 bytes 2024-09-14T05:47:51.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1413832704 bytes 2024-09-14T05:47:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1434384384 bytes 2024-09-14T05:47:51.221 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1439304704 bytes 2024-09-14T05:47:51.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1451120640 bytes 2024-09-14T05:47:51.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1507760128 bytes 2024-09-14T05:47:51.613 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1602339840 bytes 2024-09-14T05:47:51.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1648556032 bytes 2024-09-14T05:47:51.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1712421888 bytes 2024-09-14T05:47:52.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1750669312 bytes 2024-09-14T05:47:52.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1770013696 bytes 2024-09-14T05:47:52.226 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1779872768 bytes 2024-09-14T05:47:52.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1817487360 bytes 2024-09-14T05:47:52.441 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1833016320 bytes 2024-09-14T05:47:52.550 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1864925184 bytes 2024-09-14T05:47:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1909606400 bytes 2024-09-14T05:47:52.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1956382720 bytes 2024-09-14T05:47:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 1980541952 bytes 2024-09-14T05:47:53.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2025632768 bytes 2024-09-14T05:47:53.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2056622080 bytes 2024-09-14T05:47:53.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2110945280 bytes 2024-09-14T05:47:53.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2173716480 bytes 2024-09-14T05:47:53.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2229102592 bytes 2024-09-14T05:47:53.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2258381824 bytes 2024-09-14T05:47:53.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2292463616 bytes 2024-09-14T05:47:54.089 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2335982592 bytes 2024-09-14T05:47:54.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2395138048 bytes 2024-09-14T05:47:54.386 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2471324672 bytes 2024-09-14T05:47:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2529466368 bytes 2024-09-14T05:47:54.656 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2588228608 bytes 2024-09-14T05:47:54.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2600364032 bytes 2024-09-14T05:47:54.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2630490112 bytes 2024-09-14T05:47:55.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2685774848 bytes 2024-09-14T05:47:55.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2730236928 bytes 2024-09-14T05:47:55.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2758052864 bytes 2024-09-14T05:47:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2824541184 bytes 2024-09-14T05:47:55.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 2869070848 bytes 2024-09-14T05:47:55.610 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 2875672576 bytes 2024-09-14T05:47:55.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 2901455872 bytes 2024-09-14T05:47:55.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 2922184704 bytes 2024-09-14T05:47:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 2979132416 bytes 2024-09-14T05:47:55.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:47:55.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:47:55.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:47:55.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:47:55.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:47:55.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_22 2024-09-14T05:47:55.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_22 snapshot 128M 2024-09-14T05:47:55.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:47:55.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:47:55.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:47:55.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:47:55.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_22 2024-09-14T05:47:55.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:47:55.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:47:55.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-14T05:47:55.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:47:55.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_22 128M 2024-09-14T05:47:55.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:47:55.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:47:55.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:47:55.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:47:55.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_22 2024-09-14T05:47:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:47:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T05:47:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-14T05:47:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-14T05:47:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:47:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_22 2024-09-14T05:47:56.021 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T05:47:56.021 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:47:56.021 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-14T05:47:56.021 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:47:56.021 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_22 snapshot 2024-09-14T05:47:56.021 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:47:56.022 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:47:56.022 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_22 2024-09-14T05:47:56.022 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:47:56.022 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_22 snapshot 2024-09-14T05:47:56.919 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T05:47:56.929 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_22 2024-09-14T05:47:56.969 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_22': 2024-09-14T05:47:56.969 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:47:56.969 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:47:56.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T05:47:56.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 136d7689c553 2024-09-14T05:47:56.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.136d7689c553 2024-09-14T05:47:56.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:47:56.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T05:47:56.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:47:56.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:47:56.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:47:56 2024 2024-09-14T05:47:56.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:47:56 2024 2024-09-14T05:47:56.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:47:56 2024 2024-09-14T05:47:56.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:47:56.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:47:56.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 81e2e849-43ea-4a3d-825a-ccda8c67058a 2024-09-14T05:47:56.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T05:47:56.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:47:56.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_22 2024-09-14T05:47:56.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:47:56.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:47:56.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_22 2024-09-14T05:47:56.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:47:56.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=13 2024-09-14T05:47:56.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:47:56.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_22 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:47:57.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 27893760 bytes 2024-09-14T05:47:57.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 75898880 bytes 2024-09-14T05:47:57.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 86702080 bytes 2024-09-14T05:47:57.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 143440896 bytes 2024-09-14T05:47:57.656 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 218335232 bytes 2024-09-14T05:47:57.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 294457344 bytes 2024-09-14T05:47:57.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 326263808 bytes 2024-09-14T05:47:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 405438464 bytes 2024-09-14T05:47:58.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 453903360 bytes 2024-09-14T05:47:58.393 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 540858368 bytes 2024-09-14T05:47:58.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 618199040 bytes 2024-09-14T05:47:58.694 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 681954304 bytes 2024-09-14T05:47:58.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 730016768 bytes 2024-09-14T05:47:58.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 814480384 bytes 2024-09-14T05:47:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 844282880 bytes 2024-09-14T05:47:59.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 924324864 bytes 2024-09-14T05:47:59.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 949897216 bytes 2024-09-14T05:47:59.468 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 974750720 bytes 2024-09-14T05:47:59.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 1035293696 bytes 2024-09-14T05:47:59.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 1133523968 bytes 2024-09-14T05:47:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 1134150656 bytes 2024-09-14T05:47:59.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 1135756288 bytes 2024-09-14T05:48:00.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1152734208 bytes 2024-09-14T05:48:00.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1191094272 bytes 2024-09-14T05:48:00.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1226457088 bytes 2024-09-14T05:48:00.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1242584064 bytes 2024-09-14T05:48:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1294655488 bytes 2024-09-14T05:48:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1343219712 bytes 2024-09-14T05:48:00.758 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1386630144 bytes 2024-09-14T05:48:00.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1426889728 bytes 2024-09-14T05:48:01.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1520168960 bytes 2024-09-14T05:48:01.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1547554816 bytes 2024-09-14T05:48:01.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1549260800 bytes 2024-09-14T05:48:01.421 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1651124224 bytes 2024-09-14T05:48:01.514 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1666156544 bytes 2024-09-14T05:48:01.610 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1683837952 bytes 2024-09-14T05:48:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1754905600 bytes 2024-09-14T05:48:01.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1786036224 bytes 2024-09-14T05:48:02.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1826297856 bytes 2024-09-14T05:48:02.117 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1868568576 bytes 2024-09-14T05:48:02.246 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1897865216 bytes 2024-09-14T05:48:02.387 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1964465152 bytes 2024-09-14T05:48:02.502 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 2000659456 bytes 2024-09-14T05:48:02.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 2036262912 bytes 2024-09-14T05:48:02.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 2086938624 bytes 2024-09-14T05:48:02.899 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 2151284736 bytes 2024-09-14T05:48:03.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 2205385728 bytes 2024-09-14T05:48:03.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2257086464 bytes 2024-09-14T05:48:03.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2257683456 bytes 2024-09-14T05:48:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2279930880 bytes 2024-09-14T05:48:03.471 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2316738560 bytes 2024-09-14T05:48:03.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2345011200 bytes 2024-09-14T05:48:03.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2362651648 bytes 2024-09-14T05:48:03.795 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2399000576 bytes 2024-09-14T05:48:03.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2405285888 bytes 2024-09-14T05:48:03.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2426828800 bytes 2024-09-14T05:48:04.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2461977600 bytes 2024-09-14T05:48:04.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2516631552 bytes 2024-09-14T05:48:04.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2588115968 bytes 2024-09-14T05:48:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2630200320 bytes 2024-09-14T05:48:04.599 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2651016192 bytes 2024-09-14T05:48:04.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2718453760 bytes 2024-09-14T05:48:04.900 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2804641792 bytes 2024-09-14T05:48:05.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2837269504 bytes 2024-09-14T05:48:05.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2914248704 bytes 2024-09-14T05:48:05.343 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 2999501824 bytes 2024-09-14T05:48:05.452 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 3018900480 bytes 2024-09-14T05:48:05.548 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 3028210688 bytes 2024-09-14T05:48:05.693 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 3093920768 bytes 2024-09-14T05:48:05.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 3133667328 bytes 2024-09-14T05:48:05.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 3135415296 bytes 2024-09-14T05:48:06.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 3192708096 bytes 2024-09-14T05:48:06.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7300 IOs, 3213012992 bytes 2024-09-14T05:48:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7400 IOs, 3233658880 bytes 2024-09-14T05:48:06.406 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7500 IOs, 3298076672 bytes 2024-09-14T05:48:06.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7600 IOs, 3302834176 bytes 2024-09-14T05:48:06.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7700 IOs, 3350319104 bytes 2024-09-14T05:48:06.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7800 IOs, 3401918464 bytes 2024-09-14T05:48:06.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7900 IOs, 3477663744 bytes 2024-09-14T05:48:07.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8000 IOs, 3554853888 bytes 2024-09-14T05:48:07.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8100 IOs, 3555459072 bytes 2024-09-14T05:48:07.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8200 IOs, 3577013248 bytes 2024-09-14T05:48:07.401 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8300 IOs, 3631242240 bytes 2024-09-14T05:48:07.578 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8400 IOs, 3743291392 bytes 2024-09-14T05:48:07.701 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8500 IOs, 3782371328 bytes 2024-09-14T05:48:07.808 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8600 IOs, 3798451200 bytes 2024-09-14T05:48:07.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8700 IOs, 3817096192 bytes 2024-09-14T05:48:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8800 IOs, 3834626048 bytes 2024-09-14T05:48:08.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8900 IOs, 3835248640 bytes 2024-09-14T05:48:08.190 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9000 IOs, 3841054720 bytes 2024-09-14T05:48:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9100 IOs, 3862171648 bytes 2024-09-14T05:48:08.437 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9200 IOs, 3930753024 bytes 2024-09-14T05:48:08.561 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9300 IOs, 3964364800 bytes 2024-09-14T05:48:08.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9400 IOs, 4074113024 bytes 2024-09-14T05:48:08.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9500 IOs, 4104684544 bytes 2024-09-14T05:48:08.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9600 IOs, 4127416320 bytes 2024-09-14T05:48:09.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9700 IOs, 4131753984 bytes 2024-09-14T05:48:09.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9800 IOs, 4148830208 bytes 2024-09-14T05:48:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9900 IOs, 4149440512 bytes 2024-09-14T05:48:09.363 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10000 IOs, 4189735936 bytes 2024-09-14T05:48:09.545 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10100 IOs, 4277377024 bytes 2024-09-14T05:48:09.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10200 IOs, 4316961792 bytes 2024-09-14T05:48:09.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10300 IOs, 4347930624 bytes 2024-09-14T05:48:09.864 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10400 IOs, 4348539904 bytes 2024-09-14T05:48:09.966 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10500 IOs, 4373320704 bytes 2024-09-14T05:48:09.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:48:09.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:48:09.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:48:09.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:48:09.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:48:09.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_23 2024-09-14T05:48:09.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_23 snapshot 128M 2024-09-14T05:48:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:48:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:48:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:48:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:48:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_23 2024-09-14T05:48:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:48:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:48:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-14T05:48:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:48:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_23 128M 2024-09-14T05:48:09.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:48:09.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:48:09.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:48:09.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:48:09.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_23 2024-09-14T05:48:09.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:48:09.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T05:48:09.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-14T05:48:09.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-14T05:48:09.983 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:48:09.983 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_23 2024-09-14T05:48:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T05:48:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:48:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-14T05:48:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:48:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_23 snapshot 2024-09-14T05:48:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:48:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:48:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_23 2024-09-14T05:48:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:48:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_23 snapshot 2024-09-14T05:48:10.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T05:48:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_23 2024-09-14T05:48:10.998 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_23': 2024-09-14T05:48:10.998 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:48:10.998 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:48:10.998 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T05:48:10.998 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 1379988390ab 2024-09-14T05:48:10.998 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.1379988390ab 2024-09-14T05:48:10.998 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:48:10.998 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T05:48:10.998 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:48:10.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:48:10.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:48:10 2024 2024-09-14T05:48:10.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:48:10 2024 2024-09-14T05:48:10.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:48:10 2024 2024-09-14T05:48:10.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:48:10.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:48:10.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 1ad775b2-4acb-4f43-b70c-7d7c1bb4782d 2024-09-14T05:48:10.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T05:48:11.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:48:11.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_23 2024-09-14T05:48:11.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:48:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:48:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_23 2024-09-14T05:48:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:48:11.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=5 2024-09-14T05:48:11.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:48:11.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_23 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:48:11.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 66040832 bytes 2024-09-14T05:48:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 75451392 bytes 2024-09-14T05:48:11.406 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 78584832 bytes 2024-09-14T05:48:11.528 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 114876416 bytes 2024-09-14T05:48:11.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 115485696 bytes 2024-09-14T05:48:11.787 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 217449472 bytes 2024-09-14T05:48:11.899 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 254446592 bytes 2024-09-14T05:48:12.022 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 292988928 bytes 2024-09-14T05:48:12.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 319902720 bytes 2024-09-14T05:48:12.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 427684864 bytes 2024-09-14T05:48:12.424 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 450622464 bytes 2024-09-14T05:48:12.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 480106496 bytes 2024-09-14T05:48:12.626 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 493398016 bytes 2024-09-14T05:48:12.782 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 570904576 bytes 2024-09-14T05:48:12.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 617113600 bytes 2024-09-14T05:48:13.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 716608512 bytes 2024-09-14T05:48:13.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 721180672 bytes 2024-09-14T05:48:13.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 787340288 bytes 2024-09-14T05:48:13.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 825792512 bytes 2024-09-14T05:48:13.623 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 938713088 bytes 2024-09-14T05:48:13.787 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 1012497408 bytes 2024-09-14T05:48:13.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 1014702080 bytes 2024-09-14T05:48:14.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1070693376 bytes 2024-09-14T05:48:14.118 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1091805184 bytes 2024-09-14T05:48:14.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1139163136 bytes 2024-09-14T05:48:14.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1167837184 bytes 2024-09-14T05:48:14.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1227379712 bytes 2024-09-14T05:48:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1251078144 bytes 2024-09-14T05:48:14.744 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1311481856 bytes 2024-09-14T05:48:14.830 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1314124800 bytes 2024-09-14T05:48:14.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1381641216 bytes 2024-09-14T05:48:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1464203264 bytes 2024-09-14T05:48:15.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1486389248 bytes 2024-09-14T05:48:15.392 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1543764992 bytes 2024-09-14T05:48:15.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1586865152 bytes 2024-09-14T05:48:15.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1672207360 bytes 2024-09-14T05:48:15.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1738766336 bytes 2024-09-14T05:48:15.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1755961344 bytes 2024-09-14T05:48:16.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:48:16.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:48:16.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:48:16.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:48:16.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:48:16.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_24 2024-09-14T05:48:16.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_24 snapshot 128M 2024-09-14T05:48:16.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:48:16.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:48:16.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:48:16.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:48:16.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_24 2024-09-14T05:48:16.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:48:16.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:48:16.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-14T05:48:16.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:48:16.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_24 128M 2024-09-14T05:48:16.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:48:16.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:48:16.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:48:16.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:48:16.011 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_24 2024-09-14T05:48:16.011 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:48:16.011 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T05:48:16.011 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-14T05:48:16.011 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-14T05:48:16.011 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:48:16.011 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_24 2024-09-14T05:48:16.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T05:48:16.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:48:16.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-14T05:48:16.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:48:16.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_24 snapshot 2024-09-14T05:48:16.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:48:16.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:48:16.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_24 2024-09-14T05:48:16.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:48:16.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_24 snapshot 2024-09-14T05:48:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T05:48:16.962 INFO:tasks.ceph.cluster2.osd.2.smithi191.stderr:2024-09-14T05:48:16.961+0000 7f3763311640 -1 reset not still connected to 0x55facdb99040 2024-09-14T05:48:16.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_24 2024-09-14T05:48:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_24': 2024-09-14T05:48:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:48:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:48:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T05:48:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 138591009dab 2024-09-14T05:48:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.138591009dab 2024-09-14T05:48:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:48:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T05:48:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:48:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:48:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:48:16 2024 2024-09-14T05:48:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:48:16 2024 2024-09-14T05:48:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:48:16 2024 2024-09-14T05:48:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:48:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:48:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 0470f29c-7ca8-419d-8a53-5b2f3f342a4f 2024-09-14T05:48:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T05:48:17.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:48:17.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_24 2024-09-14T05:48:17.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:48:17.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:48:17.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_24 2024-09-14T05:48:17.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:48:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=10 2024-09-14T05:48:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:48:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_24 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:48:17.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 38140928 bytes 2024-09-14T05:48:17.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 75751424 bytes 2024-09-14T05:48:17.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 157537280 bytes 2024-09-14T05:48:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 171744256 bytes 2024-09-14T05:48:17.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 201920512 bytes 2024-09-14T05:48:17.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 297690112 bytes 2024-09-14T05:48:18.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 337840128 bytes 2024-09-14T05:48:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 377790464 bytes 2024-09-14T05:48:18.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 441075712 bytes 2024-09-14T05:48:18.422 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 507319296 bytes 2024-09-14T05:48:18.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 570560512 bytes 2024-09-14T05:48:18.684 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 606067712 bytes 2024-09-14T05:48:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 690158592 bytes 2024-09-14T05:48:18.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 741466112 bytes 2024-09-14T05:48:19.091 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 762820608 bytes 2024-09-14T05:48:19.247 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 840925184 bytes 2024-09-14T05:48:19.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 875737088 bytes 2024-09-14T05:48:19.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 877530112 bytes 2024-09-14T05:48:19.490 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-14T05:48:19.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-14T05:48:19.491 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-09-14T05:48:19.492 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-14T05:48:19.491+0000 7f2c792a4640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 46020) UID: 1000 2024-09-14T05:48:19.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 905224192 bytes 2024-09-14T05:48:19.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 945813504 bytes 2024-09-14T05:48:19.821 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 983609344 bytes 2024-09-14T05:48:19.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 984225792 bytes 2024-09-14T05:48:20.073 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1070064640 bytes 2024-09-14T05:48:20.172 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1087256576 bytes 2024-09-14T05:48:20.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1148004352 bytes 2024-09-14T05:48:20.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1195435008 bytes 2024-09-14T05:48:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1238260736 bytes 2024-09-14T05:48:20.681 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1269170176 bytes 2024-09-14T05:48:20.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1286585344 bytes 2024-09-14T05:48:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1312292864 bytes 2024-09-14T05:48:21.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1369373696 bytes 2024-09-14T05:48:21.161 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1422561280 bytes 2024-09-14T05:48:21.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1423168512 bytes 2024-09-14T05:48:21.340 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1438279680 bytes 2024-09-14T05:48:21.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1452094464 bytes 2024-09-14T05:48:21.562 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1495798784 bytes 2024-09-14T05:48:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1517089792 bytes 2024-09-14T05:48:21.811 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1570419712 bytes 2024-09-14T05:48:21.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1593247744 bytes 2024-09-14T05:48:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1670002688 bytes 2024-09-14T05:48:22.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1694253056 bytes 2024-09-14T05:48:22.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1702814720 bytes 2024-09-14T05:48:22.402 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1762283520 bytes 2024-09-14T05:48:22.491 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-14T05:48:22.492 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T05:48:22.492 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-14T05:48:22.492 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-14T05:48:22.492 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-14T05:48:22.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-14T05:48:22.495 INFO:tasks.rbd_mirror_thrash:waiting for 96 secs before thrashing 2024-09-14T05:48:22.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1825080320 bytes 2024-09-14T05:48:22.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1916955648 bytes 2024-09-14T05:48:22.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1939320832 bytes 2024-09-14T05:48:23.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 1988049920 bytes 2024-09-14T05:48:23.237 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2075697152 bytes 2024-09-14T05:48:23.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2113768448 bytes 2024-09-14T05:48:23.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2184679424 bytes 2024-09-14T05:48:23.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2243889152 bytes 2024-09-14T05:48:23.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2257096704 bytes 2024-09-14T05:48:23.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2312819712 bytes 2024-09-14T05:48:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2358121472 bytes 2024-09-14T05:48:24.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2386059264 bytes 2024-09-14T05:48:24.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2443824128 bytes 2024-09-14T05:48:24.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2517515264 bytes 2024-09-14T05:48:24.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2540770304 bytes 2024-09-14T05:48:24.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2566170624 bytes 2024-09-14T05:48:24.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2649397248 bytes 2024-09-14T05:48:24.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2660498432 bytes 2024-09-14T05:48:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2695883776 bytes 2024-09-14T05:48:25.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2711599104 bytes 2024-09-14T05:48:25.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2724829184 bytes 2024-09-14T05:48:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2739738624 bytes 2024-09-14T05:48:25.398 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 2740571136 bytes 2024-09-14T05:48:25.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 2778300416 bytes 2024-09-14T05:48:25.631 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 2803430400 bytes 2024-09-14T05:48:25.748 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 2835232768 bytes 2024-09-14T05:48:25.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 2879674368 bytes 2024-09-14T05:48:26.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 2940570624 bytes 2024-09-14T05:48:26.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 2954459136 bytes 2024-09-14T05:48:26.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7300 IOs, 2955087872 bytes 2024-09-14T05:48:26.322 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7400 IOs, 2978652160 bytes 2024-09-14T05:48:26.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7500 IOs, 2994844672 bytes 2024-09-14T05:48:26.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7600 IOs, 3075890176 bytes 2024-09-14T05:48:26.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7700 IOs, 3105996800 bytes 2024-09-14T05:48:26.794 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7800 IOs, 3118309376 bytes 2024-09-14T05:48:26.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7900 IOs, 3156968448 bytes 2024-09-14T05:48:27.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:48:27.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:48:27.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:48:27.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:48:27.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:48:27.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_25 2024-09-14T05:48:27.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_25 snapshot 128M 2024-09-14T05:48:27.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:48:27.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:48:27.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:48:27.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:48:27.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_25 2024-09-14T05:48:27.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:48:27.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:48:27.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-14T05:48:27.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:48:27.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_25 128M 2024-09-14T05:48:27.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:48:27.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:48:27.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:48:27.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:48:27.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_25 2024-09-14T05:48:27.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:48:27.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T05:48:27.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-14T05:48:27.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-14T05:48:27.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:48:27.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_25 2024-09-14T05:48:27.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T05:48:27.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:48:27.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-14T05:48:27.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:48:27.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_25 snapshot 2024-09-14T05:48:27.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:48:27.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:48:27.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_25 2024-09-14T05:48:27.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:48:27.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_25 snapshot 2024-09-14T05:48:27.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T05:48:27.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_25 2024-09-14T05:48:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_25': 2024-09-14T05:48:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:48:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:48:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T05:48:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 139ac92db31c 2024-09-14T05:48:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.139ac92db31c 2024-09-14T05:48:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:48:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T05:48:28.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:48:28.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:48:28.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:48:27 2024 2024-09-14T05:48:28.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:48:27 2024 2024-09-14T05:48:28.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:48:27 2024 2024-09-14T05:48:28.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:48:28.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:48:28.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: a6b55bc2-53d1-494f-9e17-76231088f41c 2024-09-14T05:48:28.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T05:48:28.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:48:28.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_25 2024-09-14T05:48:28.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:48:28.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:48:28.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_25 2024-09-14T05:48:28.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:48:28.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=9 2024-09-14T05:48:28.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:48:28.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_25 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:48:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 35553280 bytes 2024-09-14T05:48:28.352 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 71642112 bytes 2024-09-14T05:48:28.470 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 109174784 bytes 2024-09-14T05:48:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 151379968 bytes 2024-09-14T05:48:28.721 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 200098816 bytes 2024-09-14T05:48:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 241310720 bytes 2024-09-14T05:48:29.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 384106496 bytes 2024-09-14T05:48:29.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 426694656 bytes 2024-09-14T05:48:29.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 457714688 bytes 2024-09-14T05:48:29.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 487999488 bytes 2024-09-14T05:48:29.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 541404160 bytes 2024-09-14T05:48:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 609034240 bytes 2024-09-14T05:48:29.796 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 623627264 bytes 2024-09-14T05:48:29.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 656016384 bytes 2024-09-14T05:48:30.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 727245824 bytes 2024-09-14T05:48:30.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 825481216 bytes 2024-09-14T05:48:30.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 946825216 bytes 2024-09-14T05:48:30.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 986109952 bytes 2024-09-14T05:48:30.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 1018626048 bytes 2024-09-14T05:48:30.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 1071650816 bytes 2024-09-14T05:48:30.913 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 1101532160 bytes 2024-09-14T05:48:31.091 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 1195867136 bytes 2024-09-14T05:48:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1265782784 bytes 2024-09-14T05:48:31.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1337749504 bytes 2024-09-14T05:48:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1442187264 bytes 2024-09-14T05:48:31.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1492051968 bytes 2024-09-14T05:48:31.983 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1499075584 bytes 2024-09-14T05:48:32.089 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1523609600 bytes 2024-09-14T05:48:32.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1524252672 bytes 2024-09-14T05:48:32.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1607132160 bytes 2024-09-14T05:48:32.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1643497472 bytes 2024-09-14T05:48:32.561 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1663632384 bytes 2024-09-14T05:48:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1730143232 bytes 2024-09-14T05:48:32.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1775921152 bytes 2024-09-14T05:48:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1830900736 bytes 2024-09-14T05:48:33.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1865462784 bytes 2024-09-14T05:48:33.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1943284736 bytes 2024-09-14T05:48:33.370 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1999185920 bytes 2024-09-14T05:48:33.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 2023312384 bytes 2024-09-14T05:48:33.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 2087186432 bytes 2024-09-14T05:48:33.758 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 2129677312 bytes 2024-09-14T05:48:33.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 2177519616 bytes 2024-09-14T05:48:33.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 2205173760 bytes 2024-09-14T05:48:34.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 2251085824 bytes 2024-09-14T05:48:34.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 2313405440 bytes 2024-09-14T05:48:34.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 2394296320 bytes 2024-09-14T05:48:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 2448246784 bytes 2024-09-14T05:48:34.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2479773696 bytes 2024-09-14T05:48:34.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2492583936 bytes 2024-09-14T05:48:34.900 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2541868032 bytes 2024-09-14T05:48:34.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2542490624 bytes 2024-09-14T05:48:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2569888768 bytes 2024-09-14T05:48:35.229 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2616726528 bytes 2024-09-14T05:48:35.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2667939840 bytes 2024-09-14T05:48:35.451 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2679792640 bytes 2024-09-14T05:48:35.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2747677696 bytes 2024-09-14T05:48:35.705 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2770962432 bytes 2024-09-14T05:48:35.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2845133824 bytes 2024-09-14T05:48:35.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2885457920 bytes 2024-09-14T05:48:36.069 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2893186048 bytes 2024-09-14T05:48:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2965942272 bytes 2024-09-14T05:48:36.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2993755136 bytes 2024-09-14T05:48:36.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 3016985600 bytes 2024-09-14T05:48:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 3022169088 bytes 2024-09-14T05:48:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 3030376448 bytes 2024-09-14T05:48:36.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 3084762112 bytes 2024-09-14T05:48:36.900 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 3150489600 bytes 2024-09-14T05:48:37.038 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:48:37.038 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:48:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:48:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:48:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:48:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_26 2024-09-14T05:48:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_26 snapshot 128M 2024-09-14T05:48:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:48:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:48:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:48:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:48:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_26 2024-09-14T05:48:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:48:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:48:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-14T05:48:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:48:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_26 128M 2024-09-14T05:48:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:48:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:48:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:48:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:48:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_26 2024-09-14T05:48:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:48:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T05:48:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-14T05:48:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-14T05:48:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:48:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_26 2024-09-14T05:48:37.091 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T05:48:37.091 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:48:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-14T05:48:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:48:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_26 snapshot 2024-09-14T05:48:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:48:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:48:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_26 2024-09-14T05:48:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:48:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_26 snapshot 2024-09-14T05:48:37.991 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T05:48:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_26 2024-09-14T05:48:38.041 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_26': 2024-09-14T05:48:38.041 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:48:38.041 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:48:38.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T05:48:38.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 13a617d1479 2024-09-14T05:48:38.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.13a617d1479 2024-09-14T05:48:38.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:48:38.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T05:48:38.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:48:38.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:48:38.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:48:37 2024 2024-09-14T05:48:38.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:48:37 2024 2024-09-14T05:48:38.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:48:37 2024 2024-09-14T05:48:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:48:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:48:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 9154d447-2cac-4f96-97e0-0ce2dbafe588 2024-09-14T05:48:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T05:48:38.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:48:38.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_26 2024-09-14T05:48:38.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:48:38.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:48:38.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_26 2024-09-14T05:48:38.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:48:38.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=13 2024-09-14T05:48:38.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:48:38.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_26 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:48:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 30725120 bytes 2024-09-14T05:48:38.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 68512768 bytes 2024-09-14T05:48:38.477 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 109032448 bytes 2024-09-14T05:48:38.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 115599360 bytes 2024-09-14T05:48:38.720 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 179831808 bytes 2024-09-14T05:48:38.864 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 236848128 bytes 2024-09-14T05:48:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 317674496 bytes 2024-09-14T05:48:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 332601344 bytes 2024-09-14T05:48:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 405777408 bytes 2024-09-14T05:48:39.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 428225536 bytes 2024-09-14T05:48:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 489017344 bytes 2024-09-14T05:48:39.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 515872768 bytes 2024-09-14T05:48:39.784 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 579244032 bytes 2024-09-14T05:48:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 613997568 bytes 2024-09-14T05:48:40.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 640070656 bytes 2024-09-14T05:48:40.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 771961856 bytes 2024-09-14T05:48:40.381 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 845470720 bytes 2024-09-14T05:48:40.471 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 846090240 bytes 2024-09-14T05:48:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 895041536 bytes 2024-09-14T05:48:40.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 919684096 bytes 2024-09-14T05:48:40.728 INFO:tasks.ceph.cluster2.osd.1.smithi191.stderr:2024-09-14T05:48:40.727+0000 7f9375a93640 -1 reset not still connected to 0x5611e7cd25b0 2024-09-14T05:48:40.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 952256512 bytes 2024-09-14T05:48:40.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 1029383168 bytes 2024-09-14T05:48:41.100 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1057204224 bytes 2024-09-14T05:48:41.228 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1103719424 bytes 2024-09-14T05:48:41.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1156986880 bytes 2024-09-14T05:48:41.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1231050752 bytes 2024-09-14T05:48:41.631 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1260463104 bytes 2024-09-14T05:48:41.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1264233472 bytes 2024-09-14T05:48:41.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1331128320 bytes 2024-09-14T05:48:41.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1374143488 bytes 2024-09-14T05:48:42.091 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1397915648 bytes 2024-09-14T05:48:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1423970304 bytes 2024-09-14T05:48:42.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1524584448 bytes 2024-09-14T05:48:42.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1541022720 bytes 2024-09-14T05:48:42.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1568190464 bytes 2024-09-14T05:48:42.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1610000384 bytes 2024-09-14T05:48:42.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1631733760 bytes 2024-09-14T05:48:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1665792000 bytes 2024-09-14T05:48:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1809745920 bytes 2024-09-14T05:48:43.292 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1849966592 bytes 2024-09-14T05:48:43.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1888165888 bytes 2024-09-14T05:48:43.514 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1906834432 bytes 2024-09-14T05:48:43.682 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1985287168 bytes 2024-09-14T05:48:43.762 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1985915904 bytes 2024-09-14T05:48:43.865 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 2011086848 bytes 2024-09-14T05:48:43.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 2033175552 bytes 2024-09-14T05:48:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 2114624512 bytes 2024-09-14T05:48:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2195969024 bytes 2024-09-14T05:48:44.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2226385920 bytes 2024-09-14T05:48:44.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2226996224 bytes 2024-09-14T05:48:44.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2227607552 bytes 2024-09-14T05:48:44.721 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2296225792 bytes 2024-09-14T05:48:44.865 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2335053824 bytes 2024-09-14T05:48:44.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2336863232 bytes 2024-09-14T05:48:45.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2391229440 bytes 2024-09-14T05:48:45.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2420796416 bytes 2024-09-14T05:48:45.299 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2442710016 bytes 2024-09-14T05:48:45.392 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2452856832 bytes 2024-09-14T05:48:45.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2486895616 bytes 2024-09-14T05:48:45.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2563113984 bytes 2024-09-14T05:48:45.855 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2697914368 bytes 2024-09-14T05:48:45.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2698531840 bytes 2024-09-14T05:48:46.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2744733696 bytes 2024-09-14T05:48:46.087 INFO:tasks.ceph.cluster2.osd.2.smithi191.stderr:2024-09-14T05:48:46.085+0000 7f3763311640 -1 reset not still connected to 0x55facddbca90 2024-09-14T05:48:46.087 INFO:tasks.ceph.cluster2.osd.2.smithi191.stderr:2024-09-14T05:48:46.085+0000 7f3763311640 -1 reset not still connected to 0x55fae4744680 2024-09-14T05:48:46.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2755520512 bytes 2024-09-14T05:48:46.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2837895168 bytes 2024-09-14T05:48:46.440 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 2843016192 bytes 2024-09-14T05:48:46.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 2886531072 bytes 2024-09-14T05:48:46.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 2908311552 bytes 2024-09-14T05:48:46.799 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 2955782144 bytes 2024-09-14T05:48:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 2980456448 bytes 2024-09-14T05:48:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 3002156032 bytes 2024-09-14T05:48:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 3079542784 bytes 2024-09-14T05:48:47.271 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7300 IOs, 3086914560 bytes 2024-09-14T05:48:47.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7400 IOs, 3192598528 bytes 2024-09-14T05:48:47.594 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7500 IOs, 3247117312 bytes 2024-09-14T05:48:47.705 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7600 IOs, 3276827648 bytes 2024-09-14T05:48:47.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7700 IOs, 3319716864 bytes 2024-09-14T05:48:47.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7800 IOs, 3361644544 bytes 2024-09-14T05:48:48.085 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7900 IOs, 3399099392 bytes 2024-09-14T05:48:48.171 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8000 IOs, 3401544704 bytes 2024-09-14T05:48:48.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8100 IOs, 3411010560 bytes 2024-09-14T05:48:48.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8200 IOs, 3489607680 bytes 2024-09-14T05:48:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8300 IOs, 3490190336 bytes 2024-09-14T05:48:48.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8400 IOs, 3529361408 bytes 2024-09-14T05:48:48.756 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8500 IOs, 3559160832 bytes 2024-09-14T05:48:48.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8600 IOs, 3600641024 bytes 2024-09-14T05:48:49.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8700 IOs, 3636217856 bytes 2024-09-14T05:48:49.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8800 IOs, 3662181376 bytes 2024-09-14T05:48:49.198 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8900 IOs, 3670081536 bytes 2024-09-14T05:48:49.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9000 IOs, 3719675904 bytes 2024-09-14T05:48:49.470 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9100 IOs, 3789791232 bytes 2024-09-14T05:48:49.606 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9200 IOs, 3846071296 bytes 2024-09-14T05:48:49.731 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9300 IOs, 3877905408 bytes 2024-09-14T05:48:49.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9400 IOs, 3935440896 bytes 2024-09-14T05:48:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9500 IOs, 3973384192 bytes 2024-09-14T05:48:50.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9600 IOs, 3999854592 bytes 2024-09-14T05:48:50.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9700 IOs, 4053814272 bytes 2024-09-14T05:48:50.363 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9800 IOs, 4095932416 bytes 2024-09-14T05:48:50.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9900 IOs, 4096565248 bytes 2024-09-14T05:48:50.563 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10000 IOs, 4133629952 bytes 2024-09-14T05:48:50.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10100 IOs, 4234088448 bytes 2024-09-14T05:48:50.832 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10200 IOs, 4243536896 bytes 2024-09-14T05:48:50.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10300 IOs, 4285654016 bytes 2024-09-14T05:48:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:48:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:48:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:48:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:48:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:48:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_27 2024-09-14T05:48:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_27 snapshot 128M 2024-09-14T05:48:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:48:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:48:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:48:51.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:48:51.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_27 2024-09-14T05:48:51.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:48:51.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:48:51.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-14T05:48:51.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:48:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_27 128M 2024-09-14T05:48:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:48:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:48:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:48:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:48:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_27 2024-09-14T05:48:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:48:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T05:48:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-14T05:48:51.055 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-14T05:48:51.055 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:48:51.055 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_27 2024-09-14T05:48:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T05:48:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:48:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-14T05:48:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:48:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_27 snapshot 2024-09-14T05:48:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:48:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:48:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_27 2024-09-14T05:48:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:48:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_27 snapshot 2024-09-14T05:48:52.011 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T05:48:52.021 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_27 2024-09-14T05:48:52.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_27': 2024-09-14T05:48:52.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:48:52.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:48:52.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T05:48:52.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 13b2c3280aba 2024-09-14T05:48:52.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.13b2c3280aba 2024-09-14T05:48:52.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:48:52.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T05:48:52.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:48:52.062 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:48:52.062 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:48:51 2024 2024-09-14T05:48:52.062 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:48:51 2024 2024-09-14T05:48:52.062 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:48:51 2024 2024-09-14T05:48:52.062 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:48:52.062 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:48:52.062 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: c4de874f-fa8c-4df7-a048-8394cb50d034 2024-09-14T05:48:52.062 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T05:48:52.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:48:52.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_27 2024-09-14T05:48:52.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:48:52.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:48:52.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_27 2024-09-14T05:48:52.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:48:52.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=10 2024-09-14T05:48:52.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:48:52.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_27 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:48:52.451 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 20828160 bytes 2024-09-14T05:48:52.538 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 28336128 bytes 2024-09-14T05:48:52.655 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 55852032 bytes 2024-09-14T05:48:52.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 76907520 bytes 2024-09-14T05:48:52.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 107390976 bytes 2024-09-14T05:48:52.995 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 153699328 bytes 2024-09-14T05:48:53.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 207474688 bytes 2024-09-14T05:48:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 226060288 bytes 2024-09-14T05:48:53.341 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 235223040 bytes 2024-09-14T05:48:53.457 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 272166912 bytes 2024-09-14T05:48:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 337382400 bytes 2024-09-14T05:48:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 340335616 bytes 2024-09-14T05:48:53.858 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 419854336 bytes 2024-09-14T05:48:53.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 443284480 bytes 2024-09-14T05:48:54.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 513491968 bytes 2024-09-14T05:48:54.227 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 530575360 bytes 2024-09-14T05:48:54.331 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 552394752 bytes 2024-09-14T05:48:54.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 593906688 bytes 2024-09-14T05:48:54.584 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 631732224 bytes 2024-09-14T05:48:54.748 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 718382080 bytes 2024-09-14T05:48:54.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 764339200 bytes 2024-09-14T05:48:55.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 826591232 bytes 2024-09-14T05:48:55.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 869730304 bytes 2024-09-14T05:48:55.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 889824256 bytes 2024-09-14T05:48:55.380 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 928693248 bytes 2024-09-14T05:48:55.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 997278720 bytes 2024-09-14T05:48:55.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1037172736 bytes 2024-09-14T05:48:55.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1060642816 bytes 2024-09-14T05:48:55.890 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1099609088 bytes 2024-09-14T05:48:55.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1130041344 bytes 2024-09-14T05:48:56.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1235340288 bytes 2024-09-14T05:48:56.293 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1239886848 bytes 2024-09-14T05:48:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1282948096 bytes 2024-09-14T05:48:56.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1342127104 bytes 2024-09-14T05:48:56.674 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1364815872 bytes 2024-09-14T05:48:56.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1422738432 bytes 2024-09-14T05:48:56.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1512548352 bytes 2024-09-14T05:48:57.094 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1532721152 bytes 2024-09-14T05:48:57.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1562139648 bytes 2024-09-14T05:48:57.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1583852544 bytes 2024-09-14T05:48:57.437 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1612718080 bytes 2024-09-14T05:48:57.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1613320192 bytes 2024-09-14T05:48:57.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1649724416 bytes 2024-09-14T05:48:57.748 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1673583616 bytes 2024-09-14T05:48:57.858 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1698233344 bytes 2024-09-14T05:48:57.995 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1744483328 bytes 2024-09-14T05:48:58.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 1786847232 bytes 2024-09-14T05:48:58.295 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 1868309504 bytes 2024-09-14T05:48:58.376 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 1868915712 bytes 2024-09-14T05:48:58.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 1897051136 bytes 2024-09-14T05:48:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 1938436096 bytes 2024-09-14T05:48:58.692 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 1939078144 bytes 2024-09-14T05:48:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 1991405568 bytes 2024-09-14T05:48:58.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2071284736 bytes 2024-09-14T05:48:59.117 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2121508864 bytes 2024-09-14T05:48:59.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2165110784 bytes 2024-09-14T05:48:59.392 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2212755456 bytes 2024-09-14T05:48:59.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2233605120 bytes 2024-09-14T05:48:59.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2248689664 bytes 2024-09-14T05:48:59.708 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2283656192 bytes 2024-09-14T05:48:59.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2308918272 bytes 2024-09-14T05:48:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2358155264 bytes 2024-09-14T05:49:00.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2389183488 bytes 2024-09-14T05:49:00.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2428863488 bytes 2024-09-14T05:49:00.340 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2480193536 bytes 2024-09-14T05:49:00.426 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 2488786944 bytes 2024-09-14T05:49:00.557 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 2529427456 bytes 2024-09-14T05:49:00.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 2553733120 bytes 2024-09-14T05:49:00.758 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 2563263488 bytes 2024-09-14T05:49:00.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 2615693312 bytes 2024-09-14T05:49:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 2677183488 bytes 2024-09-14T05:49:01.186 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 2731967488 bytes 2024-09-14T05:49:01.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7300 IOs, 2808255488 bytes 2024-09-14T05:49:01.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7400 IOs, 2821488640 bytes 2024-09-14T05:49:01.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7500 IOs, 2842159104 bytes 2024-09-14T05:49:01.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7600 IOs, 2849809408 bytes 2024-09-14T05:49:01.787 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7700 IOs, 2913276928 bytes 2024-09-14T05:49:01.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7800 IOs, 2928236544 bytes 2024-09-14T05:49:02.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7900 IOs, 2989798400 bytes 2024-09-14T05:49:02.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:49:02.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:49:02.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:49:02.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:49:02.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:49:02.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_28 2024-09-14T05:49:02.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_28 snapshot 128M 2024-09-14T05:49:02.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:49:02.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:49:02.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:49:02.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:49:02.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_28 2024-09-14T05:49:02.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:49:02.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:49:02.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-14T05:49:02.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:49:02.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_28 128M 2024-09-14T05:49:02.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:49:02.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:49:02.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:49:02.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:49:02.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_28 2024-09-14T05:49:02.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:49:02.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T05:49:02.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-14T05:49:02.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-14T05:49:02.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:49:02.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_28 2024-09-14T05:49:02.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T05:49:02.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:49:02.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-14T05:49:02.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:49:02.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_28 snapshot 2024-09-14T05:49:02.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:49:02.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:49:02.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_28 2024-09-14T05:49:02.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:49:02.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_28 snapshot 2024-09-14T05:49:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T05:49:03.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_28 2024-09-14T05:49:03.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_28': 2024-09-14T05:49:03.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:49:03.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:49:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T05:49:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 13be93f6151a 2024-09-14T05:49:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.13be93f6151a 2024-09-14T05:49:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:49:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T05:49:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:49:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:49:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:49:02 2024 2024-09-14T05:49:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:49:02 2024 2024-09-14T05:49:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:49:02 2024 2024-09-14T05:49:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:49:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:49:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 5c71f4af-d4cf-4db0-96fd-a68b7ed67897 2024-09-14T05:49:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T05:49:03.102 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:49:03.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_28 2024-09-14T05:49:03.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:49:03.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:49:03.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_28 2024-09-14T05:49:03.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:49:03.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=11 2024-09-14T05:49:03.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:49:03.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_28 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:49:03.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 63557632 bytes 2024-09-14T05:49:03.423 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 64202752 bytes 2024-09-14T05:49:03.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 75442176 bytes 2024-09-14T05:49:03.685 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 164401152 bytes 2024-09-14T05:49:03.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 193025024 bytes 2024-09-14T05:49:03.914 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 247917568 bytes 2024-09-14T05:49:04.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 290156544 bytes 2024-09-14T05:49:04.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 321080320 bytes 2024-09-14T05:49:04.271 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 335657984 bytes 2024-09-14T05:49:04.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 396584960 bytes 2024-09-14T05:49:04.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 438124544 bytes 2024-09-14T05:49:04.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 474289152 bytes 2024-09-14T05:49:04.785 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 515507200 bytes 2024-09-14T05:49:04.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 547585024 bytes 2024-09-14T05:49:05.035 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 588866560 bytes 2024-09-14T05:49:05.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 627503104 bytes 2024-09-14T05:49:05.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 654614528 bytes 2024-09-14T05:49:05.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 699831296 bytes 2024-09-14T05:49:05.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 734818304 bytes 2024-09-14T05:49:05.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 795765760 bytes 2024-09-14T05:49:05.778 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 816023552 bytes 2024-09-14T05:49:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 828658688 bytes 2024-09-14T05:49:05.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 860278784 bytes 2024-09-14T05:49:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 897717248 bytes 2024-09-14T05:49:06.226 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 921859072 bytes 2024-09-14T05:49:06.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 964933632 bytes 2024-09-14T05:49:06.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1061468160 bytes 2024-09-14T05:49:06.699 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1133889536 bytes 2024-09-14T05:49:06.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1200237568 bytes 2024-09-14T05:49:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1233956864 bytes 2024-09-14T05:49:07.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1282979840 bytes 2024-09-14T05:49:07.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1337250816 bytes 2024-09-14T05:49:07.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1382765568 bytes 2024-09-14T05:49:07.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1410473984 bytes 2024-09-14T05:49:07.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1438467072 bytes 2024-09-14T05:49:07.749 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1472503808 bytes 2024-09-14T05:49:07.835 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1473120256 bytes 2024-09-14T05:49:08.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1518620672 bytes 2024-09-14T05:49:08.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1532525568 bytes 2024-09-14T05:49:08.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1537868800 bytes 2024-09-14T05:49:08.333 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1581504512 bytes 2024-09-14T05:49:08.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1657475072 bytes 2024-09-14T05:49:08.654 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1687154688 bytes 2024-09-14T05:49:08.770 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1725597696 bytes 2024-09-14T05:49:08.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1764402176 bytes 2024-09-14T05:49:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1779276800 bytes 2024-09-14T05:49:09.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 1832950784 bytes 2024-09-14T05:49:09.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 1891779584 bytes 2024-09-14T05:49:09.407 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 1909671936 bytes 2024-09-14T05:49:09.562 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 1955747840 bytes 2024-09-14T05:49:09.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2005280768 bytes 2024-09-14T05:49:09.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2039881728 bytes 2024-09-14T05:49:09.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2068306944 bytes 2024-09-14T05:49:10.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2113095680 bytes 2024-09-14T05:49:10.251 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2219504640 bytes 2024-09-14T05:49:10.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2236275712 bytes 2024-09-14T05:49:10.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2281501696 bytes 2024-09-14T05:49:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2342016000 bytes 2024-09-14T05:49:10.762 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2396841984 bytes 2024-09-14T05:49:10.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2419224576 bytes 2024-09-14T05:49:10.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2436768768 bytes 2024-09-14T05:49:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2459147264 bytes 2024-09-14T05:49:11.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2517243904 bytes 2024-09-14T05:49:11.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2536151040 bytes 2024-09-14T05:49:11.468 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2583380992 bytes 2024-09-14T05:49:11.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 2622086144 bytes 2024-09-14T05:49:11.712 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 2665408512 bytes 2024-09-14T05:49:11.800 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 2666035200 bytes 2024-09-14T05:49:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 2694507520 bytes 2024-09-14T05:49:12.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 2721250304 bytes 2024-09-14T05:49:12.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 2767292416 bytes 2024-09-14T05:49:12.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 2855148544 bytes 2024-09-14T05:49:12.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7300 IOs, 2947884032 bytes 2024-09-14T05:49:12.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7400 IOs, 2955838464 bytes 2024-09-14T05:49:12.719 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7500 IOs, 2983259136 bytes 2024-09-14T05:49:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7600 IOs, 3010053120 bytes 2024-09-14T05:49:12.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7700 IOs, 3074000896 bytes 2024-09-14T05:49:13.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7800 IOs, 3111876608 bytes 2024-09-14T05:49:13.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7900 IOs, 3173636096 bytes 2024-09-14T05:49:13.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8000 IOs, 3227464704 bytes 2024-09-14T05:49:13.489 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8100 IOs, 3251081216 bytes 2024-09-14T05:49:13.637 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8200 IOs, 3312698368 bytes 2024-09-14T05:49:13.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8300 IOs, 3350867968 bytes 2024-09-14T05:49:13.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8400 IOs, 3351502848 bytes 2024-09-14T05:49:13.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8500 IOs, 3388795904 bytes 2024-09-14T05:49:14.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8600 IOs, 3408285696 bytes 2024-09-14T05:49:14.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:49:14.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:49:14.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:49:14.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:49:14.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:49:14.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_29 2024-09-14T05:49:14.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_29 snapshot 128M 2024-09-14T05:49:14.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:49:14.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:49:14.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:49:14.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:49:14.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_29 2024-09-14T05:49:14.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:49:14.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:49:14.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-14T05:49:14.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:49:14.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_29 128M 2024-09-14T05:49:14.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:49:14.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:49:14.112 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:49:14.112 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:49:14.112 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_29 2024-09-14T05:49:14.112 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:49:14.112 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T05:49:14.112 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-14T05:49:14.112 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-14T05:49:14.112 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:49:14.112 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_29 2024-09-14T05:49:14.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T05:49:14.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:49:14.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-14T05:49:14.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:49:14.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_29 snapshot 2024-09-14T05:49:14.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:49:14.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:49:14.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_29 2024-09-14T05:49:14.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:49:14.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_29 snapshot 2024-09-14T05:49:15.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T05:49:15.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_29 2024-09-14T05:49:15.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_29': 2024-09-14T05:49:15.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:49:15.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:49:15.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T05:49:15.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 13ca55d0bee0 2024-09-14T05:49:15.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.13ca55d0bee0 2024-09-14T05:49:15.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:49:15.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T05:49:15.132 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:49:15.132 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:49:15.132 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:49:14 2024 2024-09-14T05:49:15.132 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:49:14 2024 2024-09-14T05:49:15.132 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:49:14 2024 2024-09-14T05:49:15.132 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:49:15.132 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:49:15.132 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 3672d66c-7b41-4af2-9094-129bf558bdc1 2024-09-14T05:49:15.132 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T05:49:15.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:49:15.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_29 2024-09-14T05:49:15.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:49:15.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:49:15.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_29 2024-09-14T05:49:15.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:49:15.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=7 2024-09-14T05:49:15.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:49:15.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_29 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:49:15.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 46917632 bytes 2024-09-14T05:49:15.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 58314752 bytes 2024-09-14T05:49:15.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 84326400 bytes 2024-09-14T05:49:15.656 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 107940864 bytes 2024-09-14T05:49:15.782 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 162983936 bytes 2024-09-14T05:49:15.890 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 178450432 bytes 2024-09-14T05:49:16.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 276322304 bytes 2024-09-14T05:49:16.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 310507520 bytes 2024-09-14T05:49:16.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 350474240 bytes 2024-09-14T05:49:16.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 460172288 bytes 2024-09-14T05:49:16.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 517466112 bytes 2024-09-14T05:49:16.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 561261568 bytes 2024-09-14T05:49:16.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 606906368 bytes 2024-09-14T05:49:16.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 628753408 bytes 2024-09-14T05:49:17.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 676477952 bytes 2024-09-14T05:49:17.293 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 751858688 bytes 2024-09-14T05:49:17.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 790231040 bytes 2024-09-14T05:49:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 889853952 bytes 2024-09-14T05:49:17.771 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 970588160 bytes 2024-09-14T05:49:17.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 1084411904 bytes 2024-09-14T05:49:18.113 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 1147116544 bytes 2024-09-14T05:49:18.225 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 1176305664 bytes 2024-09-14T05:49:18.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1213979648 bytes 2024-09-14T05:49:18.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1229569024 bytes 2024-09-14T05:49:18.556 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1252199424 bytes 2024-09-14T05:49:18.684 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1301665792 bytes 2024-09-14T05:49:18.785 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1321808896 bytes 2024-09-14T05:49:18.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1330368512 bytes 2024-09-14T05:49:19.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1367934976 bytes 2024-09-14T05:49:19.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1396549632 bytes 2024-09-14T05:49:19.273 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1488389120 bytes 2024-09-14T05:49:19.398 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1525641216 bytes 2024-09-14T05:49:19.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1598118912 bytes 2024-09-14T05:49:19.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1632239616 bytes 2024-09-14T05:49:19.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1634855936 bytes 2024-09-14T05:49:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1717822464 bytes 2024-09-14T05:49:20.089 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1790955520 bytes 2024-09-14T05:49:20.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1885426688 bytes 2024-09-14T05:49:20.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1927795712 bytes 2024-09-14T05:49:20.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1976357888 bytes 2024-09-14T05:49:20.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 2000882688 bytes 2024-09-14T05:49:20.742 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 2033561600 bytes 2024-09-14T05:49:20.889 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 2098637824 bytes 2024-09-14T05:49:21.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 2167041024 bytes 2024-09-14T05:49:21.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 2194983936 bytes 2024-09-14T05:49:21.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 2197148672 bytes 2024-09-14T05:49:21.370 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 2240914432 bytes 2024-09-14T05:49:21.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2287336448 bytes 2024-09-14T05:49:21.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2324059136 bytes 2024-09-14T05:49:21.700 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2324689920 bytes 2024-09-14T05:49:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2366033920 bytes 2024-09-14T05:49:21.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2397786112 bytes 2024-09-14T05:49:22.041 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2431483904 bytes 2024-09-14T05:49:22.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:49:22.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:49:22.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:49:22.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:49:22.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:49:22.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_30 2024-09-14T05:49:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_30 snapshot 128M 2024-09-14T05:49:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:49:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:49:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:49:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:49:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_30 2024-09-14T05:49:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:49:22.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:49:22.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-14T05:49:22.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:49:22.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_30 128M 2024-09-14T05:49:22.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:49:22.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:49:22.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:49:22.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:49:22.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_30 2024-09-14T05:49:22.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:49:22.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T05:49:22.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-14T05:49:22.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-14T05:49:22.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:49:22.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_30 2024-09-14T05:49:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T05:49:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:49:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-14T05:49:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:49:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_30 snapshot 2024-09-14T05:49:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:49:22.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:49:22.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_30 2024-09-14T05:49:22.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:49:22.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_30 snapshot 2024-09-14T05:49:23.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T05:49:23.096 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_30 2024-09-14T05:49:23.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_30': 2024-09-14T05:49:23.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:49:23.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:49:23.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T05:49:23.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 13d62aa1155b 2024-09-14T05:49:23.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.13d62aa1155b 2024-09-14T05:49:23.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:49:23.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T05:49:23.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:49:23.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:49:23.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:49:22 2024 2024-09-14T05:49:23.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:49:22 2024 2024-09-14T05:49:23.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:49:22 2024 2024-09-14T05:49:23.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:49:23.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:49:23.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 3f054947-dd92-44d7-bb13-91eab274073e 2024-09-14T05:49:23.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T05:49:23.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:49:23.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_30 2024-09-14T05:49:23.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:49:23.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:49:23.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_30 2024-09-14T05:49:23.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:49:23.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=11 2024-09-14T05:49:23.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:49:23.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_30 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:49:23.384 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 80898048 bytes 2024-09-14T05:49:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 98279424 bytes 2024-09-14T05:49:23.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 103545856 bytes 2024-09-14T05:49:23.689 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 134979584 bytes 2024-09-14T05:49:23.785 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 148457472 bytes 2024-09-14T05:49:23.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 172323840 bytes 2024-09-14T05:49:24.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 253317120 bytes 2024-09-14T05:49:24.185 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 287673344 bytes 2024-09-14T05:49:24.312 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 327613440 bytes 2024-09-14T05:49:24.489 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 412380160 bytes 2024-09-14T05:49:24.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 482665472 bytes 2024-09-14T05:49:24.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 492386304 bytes 2024-09-14T05:49:24.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 513681408 bytes 2024-09-14T05:49:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 534785024 bytes 2024-09-14T05:49:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 639264768 bytes 2024-09-14T05:49:25.272 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 692774912 bytes 2024-09-14T05:49:25.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 724347904 bytes 2024-09-14T05:49:25.546 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 801722368 bytes 2024-09-14T05:49:25.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 838177792 bytes 2024-09-14T05:49:25.787 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 875792384 bytes 2024-09-14T05:49:25.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 921412608 bytes 2024-09-14T05:49:26.070 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 984001536 bytes 2024-09-14T05:49:26.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1043378176 bytes 2024-09-14T05:49:26.328 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1073668096 bytes 2024-09-14T05:49:26.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1099422720 bytes 2024-09-14T05:49:26.471 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9436999886370349) > weight (0.25) 2024-09-14T05:49:26.471 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.48283858295922466) > weight (0.25) 2024-09-14T05:49:26.471 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.6204058007492766) > weight (0.25) 2024-09-14T05:49:26.471 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.9289308360647153) > weight (0.25) 2024-09-14T05:49:26.471 INFO:tasks.rbd_mirror_thrash:waiting for 70 secs before thrashing 2024-09-14T05:49:26.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1102727168 bytes 2024-09-14T05:49:26.636 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1128572928 bytes 2024-09-14T05:49:26.785 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1190130688 bytes 2024-09-14T05:49:26.899 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1214882816 bytes 2024-09-14T05:49:26.996 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1225053184 bytes 2024-09-14T05:49:27.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1260846080 bytes 2024-09-14T05:49:27.245 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1318371328 bytes 2024-09-14T05:49:27.393 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1382062080 bytes 2024-09-14T05:49:27.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1385906176 bytes 2024-09-14T05:49:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1439004672 bytes 2024-09-14T05:49:27.720 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1444525056 bytes 2024-09-14T05:49:27.856 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1493524480 bytes 2024-09-14T05:49:27.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1526184960 bytes 2024-09-14T05:49:28.069 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1537805312 bytes 2024-09-14T05:49:28.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1614189568 bytes 2024-09-14T05:49:28.340 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1639224320 bytes 2024-09-14T05:49:28.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1680118784 bytes 2024-09-14T05:49:28.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1702196224 bytes 2024-09-14T05:49:28.669 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1710355456 bytes 2024-09-14T05:49:28.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1736938496 bytes 2024-09-14T05:49:28.888 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1766941696 bytes 2024-09-14T05:49:28.996 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 1812467712 bytes 2024-09-14T05:49:29.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 1883566080 bytes 2024-09-14T05:49:29.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 1909727232 bytes 2024-09-14T05:49:29.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 1946579968 bytes 2024-09-14T05:49:29.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 1973065728 bytes 2024-09-14T05:49:29.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2015725568 bytes 2024-09-14T05:49:29.741 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2052644864 bytes 2024-09-14T05:49:29.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2138959872 bytes 2024-09-14T05:49:30.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2139579392 bytes 2024-09-14T05:49:30.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2187165696 bytes 2024-09-14T05:49:30.272 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2242980864 bytes 2024-09-14T05:49:30.396 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2304025600 bytes 2024-09-14T05:49:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2363885568 bytes 2024-09-14T05:49:30.668 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2382840832 bytes 2024-09-14T05:49:30.785 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2415769600 bytes 2024-09-14T05:49:30.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2431237120 bytes 2024-09-14T05:49:31.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2478228480 bytes 2024-09-14T05:49:31.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2525670400 bytes 2024-09-14T05:49:31.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2564832256 bytes 2024-09-14T05:49:31.408 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 2606283776 bytes 2024-09-14T05:49:31.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 2679079936 bytes 2024-09-14T05:49:31.691 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 2712130560 bytes 2024-09-14T05:49:31.795 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 2732870656 bytes 2024-09-14T05:49:31.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 2844336128 bytes 2024-09-14T05:49:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 2898797568 bytes 2024-09-14T05:49:32.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 2934075392 bytes 2024-09-14T05:49:32.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7300 IOs, 2973395968 bytes 2024-09-14T05:49:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7400 IOs, 3062295552 bytes 2024-09-14T05:49:32.652 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7500 IOs, 3083043840 bytes 2024-09-14T05:49:32.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7600 IOs, 3145791488 bytes 2024-09-14T05:49:32.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7700 IOs, 3209630720 bytes 2024-09-14T05:49:33.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7800 IOs, 3291896832 bytes 2024-09-14T05:49:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7900 IOs, 3388995584 bytes 2024-09-14T05:49:33.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8000 IOs, 3477673984 bytes 2024-09-14T05:49:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8100 IOs, 3514808320 bytes 2024-09-14T05:49:33.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8200 IOs, 3557010432 bytes 2024-09-14T05:49:33.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8300 IOs, 3582223360 bytes 2024-09-14T05:49:33.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8400 IOs, 3630373888 bytes 2024-09-14T05:49:34.068 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8500 IOs, 3630978048 bytes 2024-09-14T05:49:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:49:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:49:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:49:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:49:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:49:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_31 2024-09-14T05:49:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_31 snapshot 128M 2024-09-14T05:49:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:49:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:49:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:49:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:49:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_31 2024-09-14T05:49:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:49:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:49:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-14T05:49:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:49:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_31 128M 2024-09-14T05:49:34.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:49:34.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:49:34.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:49:34.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:49:34.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_31 2024-09-14T05:49:34.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:49:34.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T05:49:34.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-14T05:49:34.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-14T05:49:34.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:49:34.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_31 2024-09-14T05:49:34.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T05:49:34.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:49:34.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-14T05:49:34.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:49:34.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_31 snapshot 2024-09-14T05:49:34.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:49:34.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:49:34.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_31 2024-09-14T05:49:34.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:49:34.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_31 snapshot 2024-09-14T05:49:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T05:49:35.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_31 2024-09-14T05:49:35.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_31': 2024-09-14T05:49:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:49:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:49:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T05:49:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 13e27bc793e 2024-09-14T05:49:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.13e27bc793e 2024-09-14T05:49:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:49:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T05:49:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:49:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:49:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:49:34 2024 2024-09-14T05:49:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:49:34 2024 2024-09-14T05:49:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:49:34 2024 2024-09-14T05:49:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:49:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:49:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 0b236112-fcc0-46d0-8e57-83f45f048a34 2024-09-14T05:49:35.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T05:49:35.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:49:35.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_31 2024-09-14T05:49:35.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:49:35.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:49:35.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_31 2024-09-14T05:49:35.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:49:35.171 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=8 2024-09-14T05:49:35.171 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:49:35.171 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_31 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:49:35.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 71060480 bytes 2024-09-14T05:49:35.552 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 122207232 bytes 2024-09-14T05:49:35.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 214616064 bytes 2024-09-14T05:49:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 268806144 bytes 2024-09-14T05:49:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 330002432 bytes 2024-09-14T05:49:36.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 370875392 bytes 2024-09-14T05:49:36.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 424375296 bytes 2024-09-14T05:49:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 433895424 bytes 2024-09-14T05:49:36.514 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 489076736 bytes 2024-09-14T05:49:36.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 594105344 bytes 2024-09-14T05:49:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 625000448 bytes 2024-09-14T05:49:36.966 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 681820160 bytes 2024-09-14T05:49:37.068 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 700559360 bytes 2024-09-14T05:49:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 725986304 bytes 2024-09-14T05:49:37.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 781037568 bytes 2024-09-14T05:49:37.450 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 827164672 bytes 2024-09-14T05:49:37.561 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 856936448 bytes 2024-09-14T05:49:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 885261312 bytes 2024-09-14T05:49:37.830 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 958257152 bytes 2024-09-14T05:49:37.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 987243520 bytes 2024-09-14T05:49:38.068 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 990857216 bytes 2024-09-14T05:49:38.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 1071951872 bytes 2024-09-14T05:49:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1127205888 bytes 2024-09-14T05:49:38.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1165578240 bytes 2024-09-14T05:49:38.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1224217600 bytes 2024-09-14T05:49:38.745 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1229820928 bytes 2024-09-14T05:49:38.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1257411584 bytes 2024-09-14T05:49:38.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1299005440 bytes 2024-09-14T05:49:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1312995328 bytes 2024-09-14T05:49:39.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1324859392 bytes 2024-09-14T05:49:39.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1361153024 bytes 2024-09-14T05:49:39.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1399113728 bytes 2024-09-14T05:49:39.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1419590656 bytes 2024-09-14T05:49:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1472564224 bytes 2024-09-14T05:49:39.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1499342848 bytes 2024-09-14T05:49:39.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1522800640 bytes 2024-09-14T05:49:39.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1550341120 bytes 2024-09-14T05:49:40.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1600793600 bytes 2024-09-14T05:49:40.251 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1662944256 bytes 2024-09-14T05:49:40.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1664291840 bytes 2024-09-14T05:49:40.452 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1700861952 bytes 2024-09-14T05:49:40.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1750302720 bytes 2024-09-14T05:49:40.745 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1819186176 bytes 2024-09-14T05:49:40.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1868566528 bytes 2024-09-14T05:49:40.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1910153216 bytes 2024-09-14T05:49:41.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1923901440 bytes 2024-09-14T05:49:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 1982244864 bytes 2024-09-14T05:49:41.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2058910720 bytes 2024-09-14T05:49:41.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2110940160 bytes 2024-09-14T05:49:41.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2124681216 bytes 2024-09-14T05:49:41.724 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2138974208 bytes 2024-09-14T05:49:41.835 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2160600064 bytes 2024-09-14T05:49:42.018 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2270426112 bytes 2024-09-14T05:49:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2353546240 bytes 2024-09-14T05:49:42.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2361240576 bytes 2024-09-14T05:49:42.426 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2413396992 bytes 2024-09-14T05:49:42.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2414020608 bytes 2024-09-14T05:49:42.677 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2501570560 bytes 2024-09-14T05:49:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2595503104 bytes 2024-09-14T05:49:42.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2613280768 bytes 2024-09-14T05:49:43.089 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2669662208 bytes 2024-09-14T05:49:43.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:49:43.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:49:43.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:49:43.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:49:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:49:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_32 2024-09-14T05:49:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_32 snapshot 128M 2024-09-14T05:49:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:49:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:49:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:49:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:49:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_32 2024-09-14T05:49:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:49:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:49:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-14T05:49:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:49:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_32 128M 2024-09-14T05:49:43.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:49:43.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:49:43.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:49:43.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:49:43.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_32 2024-09-14T05:49:43.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:49:43.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T05:49:43.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-14T05:49:43.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-14T05:49:43.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:49:43.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_32 2024-09-14T05:49:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T05:49:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:49:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-14T05:49:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:49:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_32 snapshot 2024-09-14T05:49:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:49:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:49:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_32 2024-09-14T05:49:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:49:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_32 snapshot 2024-09-14T05:49:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T05:49:44.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_32 2024-09-14T05:49:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_32': 2024-09-14T05:49:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:49:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:49:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T05:49:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 13ee4df4ebc2 2024-09-14T05:49:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.13ee4df4ebc2 2024-09-14T05:49:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:49:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T05:49:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:49:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:49:44.187 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:49:43 2024 2024-09-14T05:49:44.187 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:49:43 2024 2024-09-14T05:49:44.187 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:49:43 2024 2024-09-14T05:49:44.187 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:49:44.187 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:49:44.187 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: bcdd7b18-22e7-4ae4-96b5-3c51260f96c7 2024-09-14T05:49:44.187 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T05:49:44.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:49:44.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_32 2024-09-14T05:49:44.190 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:49:44.190 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:49:44.190 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_32 2024-09-14T05:49:44.190 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:49:44.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=6 2024-09-14T05:49:44.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:49:44.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_32 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:49:44.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 83116032 bytes 2024-09-14T05:49:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 83749888 bytes 2024-09-14T05:49:44.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 174260224 bytes 2024-09-14T05:49:44.798 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 199927808 bytes 2024-09-14T05:49:44.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 231335936 bytes 2024-09-14T05:49:45.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 274949120 bytes 2024-09-14T05:49:45.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 358635520 bytes 2024-09-14T05:49:45.301 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 360011776 bytes 2024-09-14T05:49:45.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 377401344 bytes 2024-09-14T05:49:45.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 415593472 bytes 2024-09-14T05:49:45.625 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 449823744 bytes 2024-09-14T05:49:45.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 522030080 bytes 2024-09-14T05:49:45.888 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 528945152 bytes 2024-09-14T05:49:46.085 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 640329728 bytes 2024-09-14T05:49:46.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 685918208 bytes 2024-09-14T05:49:46.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 694338560 bytes 2024-09-14T05:49:46.320 INFO:tasks.ceph.cluster2.osd.1.smithi191.stderr:2024-09-14T05:49:46.317+0000 7f9375a93640 -1 reset not still connected to 0x56120003d110 2024-09-14T05:49:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 716596224 bytes 2024-09-14T05:49:46.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 728777728 bytes 2024-09-14T05:49:46.658 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 780874752 bytes 2024-09-14T05:49:46.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 795061248 bytes 2024-09-14T05:49:46.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 840972288 bytes 2024-09-14T05:49:47.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 862270464 bytes 2024-09-14T05:49:47.128 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 897809408 bytes 2024-09-14T05:49:47.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 976453632 bytes 2024-09-14T05:49:47.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1027773440 bytes 2024-09-14T05:49:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1060835328 bytes 2024-09-14T05:49:47.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1160385536 bytes 2024-09-14T05:49:47.857 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1204466688 bytes 2024-09-14T05:49:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1282144256 bytes 2024-09-14T05:49:48.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1305775104 bytes 2024-09-14T05:49:48.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1344581632 bytes 2024-09-14T05:49:48.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1356268544 bytes 2024-09-14T05:49:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1367375872 bytes 2024-09-14T05:49:48.588 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1408019456 bytes 2024-09-14T05:49:48.692 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1431915520 bytes 2024-09-14T05:49:48.857 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1513788416 bytes 2024-09-14T05:49:48.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1537908736 bytes 2024-09-14T05:49:49.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1592822784 bytes 2024-09-14T05:49:49.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1679473664 bytes 2024-09-14T05:49:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1715463168 bytes 2024-09-14T05:49:49.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1716093952 bytes 2024-09-14T05:49:49.637 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1767306240 bytes 2024-09-14T05:49:49.751 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1799319552 bytes 2024-09-14T05:49:49.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1869187072 bytes 2024-09-14T05:49:50.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1881537536 bytes 2024-09-14T05:49:50.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1917603840 bytes 2024-09-14T05:49:50.198 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:49:50.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:49:50.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:49:50.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:49:50.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:49:50.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_33 2024-09-14T05:49:50.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_33 snapshot 128M 2024-09-14T05:49:50.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:49:50.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:49:50.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:49:50.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:49:50.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_33 2024-09-14T05:49:50.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:49:50.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:49:50.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-14T05:49:50.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:49:50.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_33 128M 2024-09-14T05:49:50.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:49:50.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:49:50.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:49:50.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:49:50.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_33 2024-09-14T05:49:50.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:49:50.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T05:49:50.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-14T05:49:50.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-14T05:49:50.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:49:50.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_33 2024-09-14T05:49:50.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T05:49:50.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:49:50.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-14T05:49:50.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:49:50.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_33 snapshot 2024-09-14T05:49:50.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:49:50.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:49:50.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_33 2024-09-14T05:49:50.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:49:50.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_33 snapshot 2024-09-14T05:49:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T05:49:51.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_33 2024-09-14T05:49:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_33': 2024-09-14T05:49:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:49:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:49:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T05:49:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 13f9e5e56e3c 2024-09-14T05:49:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.13f9e5e56e3c 2024-09-14T05:49:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:49:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T05:49:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:49:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:49:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:49:50 2024 2024-09-14T05:49:51.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:49:50 2024 2024-09-14T05:49:51.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:49:50 2024 2024-09-14T05:49:51.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:49:51.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:49:51.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: abebcbd1-2502-4a74-8226-d2f7f8e8db34 2024-09-14T05:49:51.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T05:49:51.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:49:51.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_33 2024-09-14T05:49:51.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:49:51.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:49:51.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_33 2024-09-14T05:49:51.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:49:51.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=14 2024-09-14T05:49:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:49:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_33 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:49:51.424 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 66866176 bytes 2024-09-14T05:49:51.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 90858496 bytes 2024-09-14T05:49:51.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 149578752 bytes 2024-09-14T05:49:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 183317504 bytes 2024-09-14T05:49:51.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 183928832 bytes 2024-09-14T05:49:52.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 234033152 bytes 2024-09-14T05:49:52.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 273789952 bytes 2024-09-14T05:49:52.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 372585472 bytes 2024-09-14T05:49:52.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 440192000 bytes 2024-09-14T05:49:52.600 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 498762752 bytes 2024-09-14T05:49:52.778 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 596377600 bytes 2024-09-14T05:49:52.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 616102912 bytes 2024-09-14T05:49:53.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 694153216 bytes 2024-09-14T05:49:53.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 743733248 bytes 2024-09-14T05:49:53.323 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 802322432 bytes 2024-09-14T05:49:53.471 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 863420416 bytes 2024-09-14T05:49:53.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 879685632 bytes 2024-09-14T05:49:53.724 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 952209408 bytes 2024-09-14T05:49:53.832 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 984359936 bytes 2024-09-14T05:49:53.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 1064696832 bytes 2024-09-14T05:49:54.102 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 1089431552 bytes 2024-09-14T05:49:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 1129159680 bytes 2024-09-14T05:49:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1228745728 bytes 2024-09-14T05:49:54.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1254433792 bytes 2024-09-14T05:49:54.639 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1308077056 bytes 2024-09-14T05:49:54.799 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1364793344 bytes 2024-09-14T05:49:54.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1394835456 bytes 2024-09-14T05:49:55.073 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1474987008 bytes 2024-09-14T05:49:55.185 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1498873856 bytes 2024-09-14T05:49:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1547675648 bytes 2024-09-14T05:49:55.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1592398848 bytes 2024-09-14T05:49:55.601 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1667641344 bytes 2024-09-14T05:49:55.712 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1701612544 bytes 2024-09-14T05:49:55.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1743460352 bytes 2024-09-14T05:49:56.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1841710080 bytes 2024-09-14T05:49:56.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1864395776 bytes 2024-09-14T05:49:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1868639232 bytes 2024-09-14T05:49:56.295 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1872535552 bytes 2024-09-14T05:49:56.396 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1891047424 bytes 2024-09-14T05:49:56.542 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1954322432 bytes 2024-09-14T05:49:56.691 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 2021536768 bytes 2024-09-14T05:49:56.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 2072018944 bytes 2024-09-14T05:49:57.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 2179010560 bytes 2024-09-14T05:49:57.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 2206202880 bytes 2024-09-14T05:49:57.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 2237981696 bytes 2024-09-14T05:49:57.370 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 2277602304 bytes 2024-09-14T05:49:57.455 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 2278221824 bytes 2024-09-14T05:49:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2311596032 bytes 2024-09-14T05:49:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2331997184 bytes 2024-09-14T05:49:57.773 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2351047680 bytes 2024-09-14T05:49:57.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2376752128 bytes 2024-09-14T05:49:58.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2448024576 bytes 2024-09-14T05:49:58.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2478880768 bytes 2024-09-14T05:49:58.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2510823424 bytes 2024-09-14T05:49:58.399 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2567653376 bytes 2024-09-14T05:49:58.496 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6122189543702501) > weight (0.25) 2024-09-14T05:49:58.496 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.6008484508734749) > weight (0.25) 2024-09-14T05:49:58.496 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.30064715535164077) > weight (0.25) 2024-09-14T05:49:58.496 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.9121424447978653) > weight (0.25) 2024-09-14T05:49:58.497 INFO:tasks.rbd_mirror_thrash:waiting for 105 secs before thrashing 2024-09-14T05:49:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2581412864 bytes 2024-09-14T05:49:58.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2585757696 bytes 2024-09-14T05:49:58.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2645633024 bytes 2024-09-14T05:49:58.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2720919552 bytes 2024-09-14T05:49:59.094 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2847539200 bytes 2024-09-14T05:49:59.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2901151744 bytes 2024-09-14T05:49:59.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2942574592 bytes 2024-09-14T05:49:59.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 3016776704 bytes 2024-09-14T05:49:59.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 3054351360 bytes 2024-09-14T05:49:59.722 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 3064725504 bytes 2024-09-14T05:49:59.831 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 3095724032 bytes 2024-09-14T05:49:59.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 3174898688 bytes 2024-09-14T05:50:00.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 3200031744 bytes 2024-09-14T05:50:00.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 3251295232 bytes 2024-09-14T05:50:00.426 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 3347240960 bytes 2024-09-14T05:50:00.550 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 3388460032 bytes 2024-09-14T05:50:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 3397211136 bytes 2024-09-14T05:50:00.756 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7300 IOs, 3443446784 bytes 2024-09-14T05:50:00.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7400 IOs, 3523954688 bytes 2024-09-14T05:50:01.100 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7500 IOs, 3604444160 bytes 2024-09-14T05:50:01.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7600 IOs, 3619766272 bytes 2024-09-14T05:50:01.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7700 IOs, 3622005760 bytes 2024-09-14T05:50:01.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7800 IOs, 3623558144 bytes 2024-09-14T05:50:01.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7900 IOs, 3667879936 bytes 2024-09-14T05:50:01.624 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8000 IOs, 3704213504 bytes 2024-09-14T05:50:01.708 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8100 IOs, 3709076480 bytes 2024-09-14T05:50:01.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8200 IOs, 3736785920 bytes 2024-09-14T05:50:01.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8300 IOs, 3782564864 bytes 2024-09-14T05:50:02.055 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8400 IOs, 3797412864 bytes 2024-09-14T05:50:02.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8500 IOs, 3827256320 bytes 2024-09-14T05:50:02.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8600 IOs, 3874035712 bytes 2024-09-14T05:50:02.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8700 IOs, 3887917056 bytes 2024-09-14T05:50:02.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8800 IOs, 3902988288 bytes 2024-09-14T05:50:02.621 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8900 IOs, 3954185216 bytes 2024-09-14T05:50:02.721 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9000 IOs, 3970735104 bytes 2024-09-14T05:50:02.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9100 IOs, 3993223168 bytes 2024-09-14T05:50:02.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9200 IOs, 4026512384 bytes 2024-09-14T05:50:03.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9300 IOs, 4085055488 bytes 2024-09-14T05:50:03.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9400 IOs, 4129837056 bytes 2024-09-14T05:50:03.324 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9500 IOs, 4156587008 bytes 2024-09-14T05:50:03.468 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9600 IOs, 4225203200 bytes 2024-09-14T05:50:03.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9700 IOs, 4265171968 bytes 2024-09-14T05:50:03.732 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9800 IOs, 4304140288 bytes 2024-09-14T05:50:03.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9900 IOs, 4320911360 bytes 2024-09-14T05:50:03.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10000 IOs, 4378526720 bytes 2024-09-14T05:50:04.096 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10100 IOs, 4416498688 bytes 2024-09-14T05:50:04.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10200 IOs, 4480849920 bytes 2024-09-14T05:50:04.375 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10300 IOs, 4529229824 bytes 2024-09-14T05:50:04.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10400 IOs, 4573111296 bytes 2024-09-14T05:50:04.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10500 IOs, 4583856128 bytes 2024-09-14T05:50:04.728 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10600 IOs, 4631654400 bytes 2024-09-14T05:50:04.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10700 IOs, 4671870976 bytes 2024-09-14T05:50:04.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10800 IOs, 4701053952 bytes 2024-09-14T05:50:05.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10900 IOs, 4712530944 bytes 2024-09-14T05:50:05.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:11000 IOs, 4783023104 bytes 2024-09-14T05:50:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:50:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:50:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:50:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:50:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:50:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_34 2024-09-14T05:50:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_34 snapshot 128M 2024-09-14T05:50:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:50:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:50:05.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:50:05.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:50:05.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_34 2024-09-14T05:50:05.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:50:05.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:50:05.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-14T05:50:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:50:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_34 128M 2024-09-14T05:50:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:50:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:50:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:50:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:50:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_34 2024-09-14T05:50:05.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:50:05.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T05:50:05.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-14T05:50:05.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-14T05:50:05.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:50:05.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_34 2024-09-14T05:50:05.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T05:50:05.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:50:05.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-14T05:50:05.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:50:05.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_34 snapshot 2024-09-14T05:50:05.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:50:05.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:50:05.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_34 2024-09-14T05:50:05.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:50:05.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_34 snapshot 2024-09-14T05:50:05.722 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T05:50:05.732 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_34 2024-09-14T05:50:05.771 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_34': 2024-09-14T05:50:05.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:50:05.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:50:05.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T05:50:05.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 140559b3674 2024-09-14T05:50:05.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.140559b3674 2024-09-14T05:50:05.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:50:05.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T05:50:05.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:50:05.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:50:05.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:50:05 2024 2024-09-14T05:50:05.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:50:05 2024 2024-09-14T05:50:05.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:50:05 2024 2024-09-14T05:50:05.773 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:50:05.773 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:50:05.773 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 71b41da4-b167-4eb2-a0ea-67c283cfec10 2024-09-14T05:50:05.773 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T05:50:05.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:50:05.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_34 2024-09-14T05:50:05.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:50:05.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:50:05.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_34 2024-09-14T05:50:05.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:50:05.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=14 2024-09-14T05:50:05.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:50:05.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_34 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:50:06.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 88683520 bytes 2024-09-14T05:50:06.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 138336256 bytes 2024-09-14T05:50:06.264 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 152266752 bytes 2024-09-14T05:50:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 234540032 bytes 2024-09-14T05:50:06.564 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 285092864 bytes 2024-09-14T05:50:06.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 285703168 bytes 2024-09-14T05:50:06.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 304940032 bytes 2024-09-14T05:50:06.856 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 327536640 bytes 2024-09-14T05:50:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 356415488 bytes 2024-09-14T05:50:07.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 431193088 bytes 2024-09-14T05:50:07.287 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 502335488 bytes 2024-09-14T05:50:07.399 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 538060800 bytes 2024-09-14T05:50:07.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 574632960 bytes 2024-09-14T05:50:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 634508288 bytes 2024-09-14T05:50:07.798 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 665181184 bytes 2024-09-14T05:50:07.925 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 709492736 bytes 2024-09-14T05:50:08.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 739595264 bytes 2024-09-14T05:50:08.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 774605824 bytes 2024-09-14T05:50:08.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 872306688 bytes 2024-09-14T05:50:08.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 936983552 bytes 2024-09-14T05:50:08.657 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 985390080 bytes 2024-09-14T05:50:08.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 1027465216 bytes 2024-09-14T05:50:08.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1082274816 bytes 2024-09-14T05:50:09.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1113731072 bytes 2024-09-14T05:50:09.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1195995136 bytes 2024-09-14T05:50:09.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1334596608 bytes 2024-09-14T05:50:09.556 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1371648000 bytes 2024-09-14T05:50:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1412971520 bytes 2024-09-14T05:50:09.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1472190464 bytes 2024-09-14T05:50:09.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1480196096 bytes 2024-09-14T05:50:10.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1546204160 bytes 2024-09-14T05:50:10.160 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1555647488 bytes 2024-09-14T05:50:10.272 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1581195264 bytes 2024-09-14T05:50:10.502 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1736846336 bytes 2024-09-14T05:50:10.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1768210432 bytes 2024-09-14T05:50:10.762 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1819966464 bytes 2024-09-14T05:50:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1899243520 bytes 2024-09-14T05:50:11.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1939176448 bytes 2024-09-14T05:50:11.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1982784512 bytes 2024-09-14T05:50:11.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1994416128 bytes 2024-09-14T05:50:11.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 2006505472 bytes 2024-09-14T05:50:11.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 2094036992 bytes 2024-09-14T05:50:11.684 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 2137312256 bytes 2024-09-14T05:50:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 2147435520 bytes 2024-09-14T05:50:11.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 2167361536 bytes 2024-09-14T05:50:11.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 2194219008 bytes 2024-09-14T05:50:12.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 2227145728 bytes 2024-09-14T05:50:12.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2316033024 bytes 2024-09-14T05:50:12.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2382495744 bytes 2024-09-14T05:50:12.571 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2421424128 bytes 2024-09-14T05:50:12.685 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2457294848 bytes 2024-09-14T05:50:12.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2508246016 bytes 2024-09-14T05:50:12.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2508876800 bytes 2024-09-14T05:50:13.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2571537408 bytes 2024-09-14T05:50:13.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2612072448 bytes 2024-09-14T05:50:13.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2631566336 bytes 2024-09-14T05:50:13.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2647480320 bytes 2024-09-14T05:50:13.502 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2656629760 bytes 2024-09-14T05:50:13.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2684928000 bytes 2024-09-14T05:50:13.732 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2714945536 bytes 2024-09-14T05:50:13.911 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2810309632 bytes 2024-09-14T05:50:14.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2834378752 bytes 2024-09-14T05:50:14.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2871784448 bytes 2024-09-14T05:50:14.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2963877888 bytes 2024-09-14T05:50:14.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 3026045952 bytes 2024-09-14T05:50:14.615 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 3076307968 bytes 2024-09-14T05:50:14.703 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 3076916224 bytes 2024-09-14T05:50:14.844 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 3131307008 bytes 2024-09-14T05:50:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 3136480256 bytes 2024-09-14T05:50:15.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 3261186048 bytes 2024-09-14T05:50:15.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 3309786112 bytes 2024-09-14T05:50:15.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 3370897408 bytes 2024-09-14T05:50:15.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7300 IOs, 3399233536 bytes 2024-09-14T05:50:15.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7400 IOs, 3439577088 bytes 2024-09-14T05:50:15.925 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7500 IOs, 3489290240 bytes 2024-09-14T05:50:16.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7600 IOs, 3514781696 bytes 2024-09-14T05:50:16.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7700 IOs, 3544976384 bytes 2024-09-14T05:50:16.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7800 IOs, 3549225984 bytes 2024-09-14T05:50:16.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7900 IOs, 3564299264 bytes 2024-09-14T05:50:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8000 IOs, 3576253440 bytes 2024-09-14T05:50:16.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8100 IOs, 3600382976 bytes 2024-09-14T05:50:16.730 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8200 IOs, 3699822592 bytes 2024-09-14T05:50:16.836 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8300 IOs, 3720567808 bytes 2024-09-14T05:50:16.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8400 IOs, 3803505664 bytes 2024-09-14T05:50:17.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8500 IOs, 3830504448 bytes 2024-09-14T05:50:17.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8600 IOs, 3831792640 bytes 2024-09-14T05:50:17.288 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8700 IOs, 3844732928 bytes 2024-09-14T05:50:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8800 IOs, 3852659712 bytes 2024-09-14T05:50:17.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8900 IOs, 3869131776 bytes 2024-09-14T05:50:17.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9000 IOs, 3909729280 bytes 2024-09-14T05:50:17.926 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9100 IOs, 3945037824 bytes 2024-09-14T05:50:18.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9200 IOs, 3945646080 bytes 2024-09-14T05:50:18.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9300 IOs, 4022641664 bytes 2024-09-14T05:50:18.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9400 IOs, 4032735232 bytes 2024-09-14T05:50:18.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9500 IOs, 4053091328 bytes 2024-09-14T05:50:18.499 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9600 IOs, 4100208640 bytes 2024-09-14T05:50:18.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9700 IOs, 4147798016 bytes 2024-09-14T05:50:18.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9800 IOs, 4243701760 bytes 2024-09-14T05:50:18.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9900 IOs, 4261700608 bytes 2024-09-14T05:50:19.041 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10000 IOs, 4309504000 bytes 2024-09-14T05:50:19.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10100 IOs, 4323184640 bytes 2024-09-14T05:50:19.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10200 IOs, 4407065600 bytes 2024-09-14T05:50:19.459 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10300 IOs, 4460143616 bytes 2024-09-14T05:50:19.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10400 IOs, 4523663360 bytes 2024-09-14T05:50:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10500 IOs, 4593329152 bytes 2024-09-14T05:50:19.782 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:50:19.782 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:50:19.782 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:50:19.782 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:50:19.782 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:50:19.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_35 2024-09-14T05:50:19.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_35 snapshot 128M 2024-09-14T05:50:19.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:50:19.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:50:19.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:50:19.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:50:19.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_35 2024-09-14T05:50:19.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:50:19.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:50:19.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-14T05:50:19.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:50:19.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_35 128M 2024-09-14T05:50:19.784 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:50:19.784 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:50:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:50:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:50:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_35 2024-09-14T05:50:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:50:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T05:50:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-14T05:50:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-14T05:50:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:50:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_35 2024-09-14T05:50:19.835 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T05:50:19.835 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:50:19.835 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-14T05:50:19.835 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:50:19.835 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_35 snapshot 2024-09-14T05:50:19.835 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:50:19.836 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:50:19.836 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_35 2024-09-14T05:50:19.836 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:50:19.836 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_35 snapshot 2024-09-14T05:50:20.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T05:50:20.096 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_35 2024-09-14T05:50:20.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_35': 2024-09-14T05:50:20.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:50:20.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:50:20.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T05:50:20.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 14119e229dae 2024-09-14T05:50:20.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.14119e229dae 2024-09-14T05:50:20.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:50:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T05:50:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:50:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:50:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:50:19 2024 2024-09-14T05:50:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:50:19 2024 2024-09-14T05:50:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:50:19 2024 2024-09-14T05:50:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:50:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:50:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 6249a2fd-1afb-4dee-89ca-39ba30cbe74d 2024-09-14T05:50:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T05:50:20.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:50:20.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_35 2024-09-14T05:50:20.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:50:20.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:50:20.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_35 2024-09-14T05:50:20.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:50:20.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=10 2024-09-14T05:50:20.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:50:20.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_35 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:50:20.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 101737472 bytes 2024-09-14T05:50:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 126219264 bytes 2024-09-14T05:50:20.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 170359808 bytes 2024-09-14T05:50:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 217279488 bytes 2024-09-14T05:50:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 240980992 bytes 2024-09-14T05:50:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 326589440 bytes 2024-09-14T05:50:21.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 367661056 bytes 2024-09-14T05:50:21.286 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 391102464 bytes 2024-09-14T05:50:21.407 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 420267008 bytes 2024-09-14T05:50:21.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 421741568 bytes 2024-09-14T05:50:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 472794112 bytes 2024-09-14T05:50:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 525938688 bytes 2024-09-14T05:50:21.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 546999296 bytes 2024-09-14T05:50:21.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 568706048 bytes 2024-09-14T05:50:22.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 586654720 bytes 2024-09-14T05:50:22.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 629275648 bytes 2024-09-14T05:50:22.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 681029632 bytes 2024-09-14T05:50:22.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 756088832 bytes 2024-09-14T05:50:22.689 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 846251008 bytes 2024-09-14T05:50:22.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 918945792 bytes 2024-09-14T05:50:22.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 930772992 bytes 2024-09-14T05:50:23.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 952163328 bytes 2024-09-14T05:50:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1021296640 bytes 2024-09-14T05:50:23.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1060469760 bytes 2024-09-14T05:50:23.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1151709184 bytes 2024-09-14T05:50:23.673 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1228743680 bytes 2024-09-14T05:50:23.821 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1296844800 bytes 2024-09-14T05:50:23.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1366351872 bytes 2024-09-14T05:50:24.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1406537728 bytes 2024-09-14T05:50:24.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1468944384 bytes 2024-09-14T05:50:24.374 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1493143552 bytes 2024-09-14T05:50:24.468 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1498811392 bytes 2024-09-14T05:50:24.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1587680256 bytes 2024-09-14T05:50:24.782 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1646593024 bytes 2024-09-14T05:50:24.900 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1676678144 bytes 2024-09-14T05:50:25.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1733857280 bytes 2024-09-14T05:50:25.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1757493248 bytes 2024-09-14T05:50:25.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1782626304 bytes 2024-09-14T05:50:25.343 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1783228416 bytes 2024-09-14T05:50:25.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1839696896 bytes 2024-09-14T05:50:25.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1871668224 bytes 2024-09-14T05:50:25.724 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1909847040 bytes 2024-09-14T05:50:25.900 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1995642880 bytes 2024-09-14T05:50:25.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 2009902080 bytes 2024-09-14T05:50:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 2050017280 bytes 2024-09-14T05:50:26.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 2079730688 bytes 2024-09-14T05:50:26.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 2099168256 bytes 2024-09-14T05:50:26.450 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2125609984 bytes 2024-09-14T05:50:26.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2172715008 bytes 2024-09-14T05:50:26.719 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2214098944 bytes 2024-09-14T05:50:26.857 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2259180544 bytes 2024-09-14T05:50:27.019 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2343907328 bytes 2024-09-14T05:50:27.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2362507264 bytes 2024-09-14T05:50:27.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2398792704 bytes 2024-09-14T05:50:27.383 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2462263296 bytes 2024-09-14T05:50:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2487158784 bytes 2024-09-14T05:50:27.612 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2513835008 bytes 2024-09-14T05:50:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2564682752 bytes 2024-09-14T05:50:27.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2602643456 bytes 2024-09-14T05:50:27.966 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2621451264 bytes 2024-09-14T05:50:28.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2658099200 bytes 2024-09-14T05:50:28.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2733663232 bytes 2024-09-14T05:50:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2765496320 bytes 2024-09-14T05:50:28.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2818232320 bytes 2024-09-14T05:50:28.673 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2895020032 bytes 2024-09-14T05:50:28.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 2942141440 bytes 2024-09-14T05:50:28.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 2993203200 bytes 2024-09-14T05:50:29.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 3014637568 bytes 2024-09-14T05:50:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 3036152832 bytes 2024-09-14T05:50:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 3103712256 bytes 2024-09-14T05:50:29.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 3104645120 bytes 2024-09-14T05:50:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 3190135808 bytes 2024-09-14T05:50:29.696 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7300 IOs, 3237281792 bytes 2024-09-14T05:50:29.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7400 IOs, 3289454592 bytes 2024-09-14T05:50:29.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7500 IOs, 3301960704 bytes 2024-09-14T05:50:30.051 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7600 IOs, 3338688512 bytes 2024-09-14T05:50:30.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:50:30.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:50:30.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:50:30.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:50:30.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:50:30.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_36 2024-09-14T05:50:30.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_36 snapshot 128M 2024-09-14T05:50:30.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:50:30.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:50:30.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:50:30.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:50:30.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_36 2024-09-14T05:50:30.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:50:30.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:50:30.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-14T05:50:30.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:50:30.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_36 128M 2024-09-14T05:50:30.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:50:30.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:50:30.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:50:30.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:50:30.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_36 2024-09-14T05:50:30.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:50:30.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T05:50:30.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-14T05:50:30.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-14T05:50:30.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:50:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_36 2024-09-14T05:50:30.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T05:50:30.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:50:30.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-14T05:50:30.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:50:30.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_36 snapshot 2024-09-14T05:50:30.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:50:30.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:50:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_36 2024-09-14T05:50:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:50:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_36 snapshot 2024-09-14T05:50:31.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T05:50:31.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_36 2024-09-14T05:50:31.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_36': 2024-09-14T05:50:31.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:50:31.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:50:31.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T05:50:31.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 141de9775cc4 2024-09-14T05:50:31.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.141de9775cc4 2024-09-14T05:50:31.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:50:31.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T05:50:31.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:50:31.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:50:31.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:50:30 2024 2024-09-14T05:50:31.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:50:30 2024 2024-09-14T05:50:31.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:50:30 2024 2024-09-14T05:50:31.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:50:31.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:50:31.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 702a468e-cfaa-4567-a91a-60fa43b31b1f 2024-09-14T05:50:31.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T05:50:31.185 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:50:31.185 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_36 2024-09-14T05:50:31.186 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:50:31.186 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:50:31.186 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_36 2024-09-14T05:50:31.186 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:50:31.187 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=5 2024-09-14T05:50:31.187 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:50:31.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_36 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:50:31.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 4014080 bytes 2024-09-14T05:50:31.468 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 27047936 bytes 2024-09-14T05:50:31.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 124252160 bytes 2024-09-14T05:50:31.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 233646080 bytes 2024-09-14T05:50:31.965 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 268323840 bytes 2024-09-14T05:50:32.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 300986368 bytes 2024-09-14T05:50:32.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 313991168 bytes 2024-09-14T05:50:32.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 331304960 bytes 2024-09-14T05:50:32.398 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 366864384 bytes 2024-09-14T05:50:32.514 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 400900096 bytes 2024-09-14T05:50:32.656 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 448804864 bytes 2024-09-14T05:50:32.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 508952576 bytes 2024-09-14T05:50:32.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 588840960 bytes 2024-09-14T05:50:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 590808064 bytes 2024-09-14T05:50:33.161 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 605661184 bytes 2024-09-14T05:50:33.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 705443840 bytes 2024-09-14T05:50:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 716237824 bytes 2024-09-14T05:50:33.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 751544320 bytes 2024-09-14T05:50:33.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 779536384 bytes 2024-09-14T05:50:33.808 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 823289856 bytes 2024-09-14T05:50:33.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 866246656 bytes 2024-09-14T05:50:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 886984704 bytes 2024-09-14T05:50:34.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 923486208 bytes 2024-09-14T05:50:34.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 970169344 bytes 2024-09-14T05:50:34.406 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 994651136 bytes 2024-09-14T05:50:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1023187968 bytes 2024-09-14T05:50:34.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1027805184 bytes 2024-09-14T05:50:34.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1095233536 bytes 2024-09-14T05:50:34.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1156147200 bytes 2024-09-14T05:50:35.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1175024640 bytes 2024-09-14T05:50:35.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1217934336 bytes 2024-09-14T05:50:35.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1218539520 bytes 2024-09-14T05:50:35.333 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1241050112 bytes 2024-09-14T05:50:35.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1272949760 bytes 2024-09-14T05:50:35.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1289578496 bytes 2024-09-14T05:50:35.668 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1328303104 bytes 2024-09-14T05:50:35.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1348186112 bytes 2024-09-14T05:50:35.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1386897408 bytes 2024-09-14T05:50:36.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1409728512 bytes 2024-09-14T05:50:36.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1493051392 bytes 2024-09-14T05:50:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:50:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:50:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:50:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:50:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:50:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_37 2024-09-14T05:50:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_37 snapshot 128M 2024-09-14T05:50:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:50:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:50:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:50:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:50:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_37 2024-09-14T05:50:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:50:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:50:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-14T05:50:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:50:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_37 128M 2024-09-14T05:50:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:50:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:50:36.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:50:36.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:50:36.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_37 2024-09-14T05:50:36.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:50:36.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T05:50:36.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-14T05:50:36.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-14T05:50:36.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:50:36.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_37 2024-09-14T05:50:36.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T05:50:36.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:50:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-14T05:50:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:50:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_37 snapshot 2024-09-14T05:50:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:50:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:50:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_37 2024-09-14T05:50:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:50:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_37 snapshot 2024-09-14T05:50:36.472 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8980960188093742) > weight (0.25) 2024-09-14T05:50:36.472 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-09-14T05:50:36.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-09-14T05:50:36.473 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-09-14T05:50:36.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:2024-09-14T05:50:36.473+0000 7f2bce9dd640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 36698) UID: 1000 2024-09-14T05:50:37.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T05:50:37.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_37 2024-09-14T05:50:37.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_37': 2024-09-14T05:50:37.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:50:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:50:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T05:50:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 14297dcd3caa 2024-09-14T05:50:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.14297dcd3caa 2024-09-14T05:50:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:50:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T05:50:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:50:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:50:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:50:36 2024 2024-09-14T05:50:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:50:36 2024 2024-09-14T05:50:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:50:36 2024 2024-09-14T05:50:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:50:37.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:50:37.193 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 07ff7849-860f-4f87-8bed-88f824442785 2024-09-14T05:50:37.193 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T05:50:37.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:50:37.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_37 2024-09-14T05:50:37.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:50:37.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:50:37.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_37 2024-09-14T05:50:37.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:50:37.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=5 2024-09-14T05:50:37.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:50:37.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_37 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:50:37.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 39424000 bytes 2024-09-14T05:50:37.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 68485120 bytes 2024-09-14T05:50:37.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 143128576 bytes 2024-09-14T05:50:37.800 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 190137344 bytes 2024-09-14T05:50:37.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 263840768 bytes 2024-09-14T05:50:38.088 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 329511936 bytes 2024-09-14T05:50:38.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 383295488 bytes 2024-09-14T05:50:38.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 508138496 bytes 2024-09-14T05:50:38.557 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 540580864 bytes 2024-09-14T05:50:38.708 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 586033152 bytes 2024-09-14T05:50:38.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 694815744 bytes 2024-09-14T05:50:39.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 749293568 bytes 2024-09-14T05:50:39.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 812890112 bytes 2024-09-14T05:50:39.296 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 850727936 bytes 2024-09-14T05:50:39.408 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 882810880 bytes 2024-09-14T05:50:39.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 905031680 bytes 2024-09-14T05:50:39.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 962710528 bytes 2024-09-14T05:50:39.776 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 1009206272 bytes 2024-09-14T05:50:39.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 1040513024 bytes 2024-09-14T05:50:40.035 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 1101506560 bytes 2024-09-14T05:50:40.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 1109470208 bytes 2024-09-14T05:50:40.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 1138504704 bytes 2024-09-14T05:50:40.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1148499968 bytes 2024-09-14T05:50:40.498 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1250738176 bytes 2024-09-14T05:50:40.626 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1286510592 bytes 2024-09-14T05:50:40.737 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1311304704 bytes 2024-09-14T05:50:40.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1349562368 bytes 2024-09-14T05:50:40.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1392913408 bytes 2024-09-14T05:50:41.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1428741120 bytes 2024-09-14T05:50:41.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1452081152 bytes 2024-09-14T05:50:41.329 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1481535488 bytes 2024-09-14T05:50:41.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1504471040 bytes 2024-09-14T05:50:41.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1519920128 bytes 2024-09-14T05:50:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1567400960 bytes 2024-09-14T05:50:41.791 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1611057152 bytes 2024-09-14T05:50:41.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1645528064 bytes 2024-09-14T05:50:42.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1688994816 bytes 2024-09-14T05:50:42.119 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1692419072 bytes 2024-09-14T05:50:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:50:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:50:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:50:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:50:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:50:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_38 2024-09-14T05:50:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_38 snapshot 128M 2024-09-14T05:50:42.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:50:42.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:50:42.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:50:42.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:50:42.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_38 2024-09-14T05:50:42.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:50:42.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:50:42.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-14T05:50:42.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:50:42.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_38 128M 2024-09-14T05:50:42.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:50:42.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:50:42.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:50:42.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:50:42.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_38 2024-09-14T05:50:42.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:50:42.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T05:50:42.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-14T05:50:42.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-14T05:50:42.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:50:42.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_38 2024-09-14T05:50:42.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T05:50:42.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:50:42.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-14T05:50:42.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:50:42.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_38 snapshot 2024-09-14T05:50:42.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:50:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:50:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_38 2024-09-14T05:50:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:50:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_38 snapshot 2024-09-14T05:50:43.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T05:50:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_38 2024-09-14T05:50:43.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_38': 2024-09-14T05:50:43.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:50:43.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:50:43.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T05:50:43.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 14355b97e0ef 2024-09-14T05:50:43.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.14355b97e0ef 2024-09-14T05:50:43.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:50:43.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T05:50:43.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:50:43.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:50:43.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:50:42 2024 2024-09-14T05:50:43.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:50:42 2024 2024-09-14T05:50:43.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:50:42 2024 2024-09-14T05:50:43.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:50:43.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:50:43.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 4cab209c-76f4-435e-9f27-921251cc9286 2024-09-14T05:50:43.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T05:50:43.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:50:43.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_38 2024-09-14T05:50:43.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:50:43.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:50:43.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_38 2024-09-14T05:50:43.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:50:43.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=8 2024-09-14T05:50:43.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:50:43.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_38 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:50:43.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 68925440 bytes 2024-09-14T05:50:43.545 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 79172608 bytes 2024-09-14T05:50:43.683 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 135050240 bytes 2024-09-14T05:50:43.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 141315072 bytes 2024-09-14T05:50:43.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 200751104 bytes 2024-09-14T05:50:44.056 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 257934336 bytes 2024-09-14T05:50:44.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 298707968 bytes 2024-09-14T05:50:44.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 319913984 bytes 2024-09-14T05:50:44.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 343538688 bytes 2024-09-14T05:50:44.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 363809792 bytes 2024-09-14T05:50:44.657 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 445761536 bytes 2024-09-14T05:50:44.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 491677696 bytes 2024-09-14T05:50:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 497585152 bytes 2024-09-14T05:50:44.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 508624896 bytes 2024-09-14T05:50:45.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 587971584 bytes 2024-09-14T05:50:45.251 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 610958336 bytes 2024-09-14T05:50:45.370 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 643293184 bytes 2024-09-14T05:50:45.473 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-09-14T05:50:45.474 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T05:50:45.474 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-09-14T05:50:45.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-14T05:50:45.475 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-14T05:50:45.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-14T05:50:45.481 INFO:tasks.rbd_mirror_thrash:waiting for 86 secs before thrashing 2024-09-14T05:50:45.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 695087104 bytes 2024-09-14T05:50:45.673 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 768673792 bytes 2024-09-14T05:50:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 802771968 bytes 2024-09-14T05:50:45.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 864916480 bytes 2024-09-14T05:50:46.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 926784512 bytes 2024-09-14T05:50:46.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 982962176 bytes 2024-09-14T05:50:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1023650816 bytes 2024-09-14T05:50:46.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1087253504 bytes 2024-09-14T05:50:46.659 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1158368256 bytes 2024-09-14T05:50:46.830 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1219573760 bytes 2024-09-14T05:50:46.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1301312512 bytes 2024-09-14T05:50:47.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1350634496 bytes 2024-09-14T05:50:47.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1436107776 bytes 2024-09-14T05:50:47.436 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1483676672 bytes 2024-09-14T05:50:47.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1545846784 bytes 2024-09-14T05:50:47.692 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1563548672 bytes 2024-09-14T05:50:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1590290432 bytes 2024-09-14T05:50:47.951 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1665671168 bytes 2024-09-14T05:50:48.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1680277504 bytes 2024-09-14T05:50:48.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1683504128 bytes 2024-09-14T05:50:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1778920448 bytes 2024-09-14T05:50:48.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1827078144 bytes 2024-09-14T05:50:48.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1870349312 bytes 2024-09-14T05:50:48.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1908947968 bytes 2024-09-14T05:50:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1979929600 bytes 2024-09-14T05:50:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1980578816 bytes 2024-09-14T05:50:49.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1997066240 bytes 2024-09-14T05:50:49.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1997675520 bytes 2024-09-14T05:50:49.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 2007560192 bytes 2024-09-14T05:50:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 2031684608 bytes 2024-09-14T05:50:49.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2093403136 bytes 2024-09-14T05:50:49.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2141586432 bytes 2024-09-14T05:50:49.797 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2221804544 bytes 2024-09-14T05:50:49.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2271221760 bytes 2024-09-14T05:50:50.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2279112704 bytes 2024-09-14T05:50:50.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2353852416 bytes 2024-09-14T05:50:50.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2378819584 bytes 2024-09-14T05:50:50.408 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2404696064 bytes 2024-09-14T05:50:50.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2435802112 bytes 2024-09-14T05:50:50.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2495060992 bytes 2024-09-14T05:50:50.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2548132864 bytes 2024-09-14T05:50:50.913 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2572850176 bytes 2024-09-14T05:50:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2650449920 bytes 2024-09-14T05:50:51.185 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2666903552 bytes 2024-09-14T05:50:51.221 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:50:51.221 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:50:51.221 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:50:51.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:50:51.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:50:51.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_39 2024-09-14T05:50:51.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_39 snapshot 128M 2024-09-14T05:50:51.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:50:51.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:50:51.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:50:51.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:50:51.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_39 2024-09-14T05:50:51.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:50:51.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:50:51.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-14T05:50:51.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:50:51.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_39 128M 2024-09-14T05:50:51.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:50:51.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:50:51.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:50:51.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:50:51.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_39 2024-09-14T05:50:51.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:50:51.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T05:50:51.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-14T05:50:51.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-14T05:50:51.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:50:51.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_39 2024-09-14T05:50:51.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T05:50:51.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:50:51.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-14T05:50:51.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:50:51.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_39 snapshot 2024-09-14T05:50:51.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:50:51.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:50:51.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_39 2024-09-14T05:50:51.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:50:51.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_39 snapshot 2024-09-14T05:50:51.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T05:50:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_39 2024-09-14T05:50:51.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_39': 2024-09-14T05:50:51.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:50:51.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:50:51.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T05:50:51.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 1446d03e8fdc 2024-09-14T05:50:51.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.1446d03e8fdc 2024-09-14T05:50:51.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:50:51.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T05:50:51.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:50:51.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:50:51.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:50:51 2024 2024-09-14T05:50:51.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:50:51 2024 2024-09-14T05:50:51.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:50:51 2024 2024-09-14T05:50:51.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:50:51.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:50:51.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 37b3b339-d8a4-4dd0-ac91-56b0d1df8ebd 2024-09-14T05:50:51.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T05:50:51.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:50:51.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_39 2024-09-14T05:50:51.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:50:51.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:50:51.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_39 2024-09-14T05:50:51.778 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:50:51.779 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=13 2024-09-14T05:50:51.779 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:50:51.779 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_39 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:50:52.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 56099840 bytes 2024-09-14T05:50:52.094 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 63546368 bytes 2024-09-14T05:50:52.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 69096448 bytes 2024-09-14T05:50:52.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 111916032 bytes 2024-09-14T05:50:52.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 168632320 bytes 2024-09-14T05:50:52.577 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 215021568 bytes 2024-09-14T05:50:52.701 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 255611904 bytes 2024-09-14T05:50:52.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 297916416 bytes 2024-09-14T05:50:52.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 315217920 bytes 2024-09-14T05:50:53.035 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 328135680 bytes 2024-09-14T05:50:53.288 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 431767552 bytes 2024-09-14T05:50:53.422 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 496929792 bytes 2024-09-14T05:50:53.546 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 516312064 bytes 2024-09-14T05:50:53.658 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 539128832 bytes 2024-09-14T05:50:53.781 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 575769600 bytes 2024-09-14T05:50:53.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 612015104 bytes 2024-09-14T05:50:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 643660800 bytes 2024-09-14T05:50:54.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 659267584 bytes 2024-09-14T05:50:54.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 743152640 bytes 2024-09-14T05:50:54.400 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 773157888 bytes 2024-09-14T05:50:54.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 779940864 bytes 2024-09-14T05:50:54.623 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 837089280 bytes 2024-09-14T05:50:54.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 888437760 bytes 2024-09-14T05:50:54.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 914039808 bytes 2024-09-14T05:50:55.062 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 939588608 bytes 2024-09-14T05:50:55.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 984340480 bytes 2024-09-14T05:50:55.325 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1032079360 bytes 2024-09-14T05:50:55.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1034311680 bytes 2024-09-14T05:50:55.549 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1034919936 bytes 2024-09-14T05:50:55.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1109297152 bytes 2024-09-14T05:50:55.851 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1175998464 bytes 2024-09-14T05:50:55.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1191582720 bytes 2024-09-14T05:50:56.068 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1221522432 bytes 2024-09-14T05:50:56.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1241319424 bytes 2024-09-14T05:50:56.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1255463936 bytes 2024-09-14T05:50:56.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1300424704 bytes 2024-09-14T05:50:56.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1381084160 bytes 2024-09-14T05:50:56.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1396450304 bytes 2024-09-14T05:50:56.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1481285632 bytes 2024-09-14T05:50:56.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1524896768 bytes 2024-09-14T05:50:57.102 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1565971456 bytes 2024-09-14T05:50:57.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1592660992 bytes 2024-09-14T05:50:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1651447808 bytes 2024-09-14T05:50:57.477 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1679007744 bytes 2024-09-14T05:50:57.601 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1708783616 bytes 2024-09-14T05:50:57.742 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1784028160 bytes 2024-09-14T05:50:57.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 1878921216 bytes 2024-09-14T05:50:58.068 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 1925245952 bytes 2024-09-14T05:50:58.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 1986679808 bytes 2024-09-14T05:50:58.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2032614400 bytes 2024-09-14T05:50:58.441 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2060060672 bytes 2024-09-14T05:50:58.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2111721472 bytes 2024-09-14T05:50:58.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2191443968 bytes 2024-09-14T05:50:58.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2257447936 bytes 2024-09-14T05:50:59.038 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2303503360 bytes 2024-09-14T05:50:59.226 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2407520256 bytes 2024-09-14T05:50:59.326 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2427130880 bytes 2024-09-14T05:50:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2442293248 bytes 2024-09-14T05:50:59.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2472892416 bytes 2024-09-14T05:50:59.684 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2529854464 bytes 2024-09-14T05:50:59.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2541780992 bytes 2024-09-14T05:50:59.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2582702080 bytes 2024-09-14T05:51:00.054 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2631281664 bytes 2024-09-14T05:51:00.160 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2652600320 bytes 2024-09-14T05:51:00.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2678290432 bytes 2024-09-14T05:51:00.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 2739239936 bytes 2024-09-14T05:51:00.571 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 2809300992 bytes 2024-09-14T05:51:00.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 2812521472 bytes 2024-09-14T05:51:00.784 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 2850268160 bytes 2024-09-14T05:51:00.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 2894002176 bytes 2024-09-14T05:51:01.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 2945545216 bytes 2024-09-14T05:51:01.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 3022743552 bytes 2024-09-14T05:51:01.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7300 IOs, 3073074176 bytes 2024-09-14T05:51:01.455 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7400 IOs, 3110042624 bytes 2024-09-14T05:51:01.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7500 IOs, 3129030656 bytes 2024-09-14T05:51:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7600 IOs, 3176039424 bytes 2024-09-14T05:51:01.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7700 IOs, 3193320448 bytes 2024-09-14T05:51:01.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7800 IOs, 3229815808 bytes 2024-09-14T05:51:02.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7900 IOs, 3249533952 bytes 2024-09-14T05:51:02.118 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8000 IOs, 3269700608 bytes 2024-09-14T05:51:02.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8100 IOs, 3294455808 bytes 2024-09-14T05:51:02.341 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8200 IOs, 3321175040 bytes 2024-09-14T05:51:02.450 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8300 IOs, 3340740608 bytes 2024-09-14T05:51:02.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8400 IOs, 3428318208 bytes 2024-09-14T05:51:02.776 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8500 IOs, 3499787264 bytes 2024-09-14T05:51:02.890 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8600 IOs, 3533671424 bytes 2024-09-14T05:51:03.019 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8700 IOs, 3572250624 bytes 2024-09-14T05:51:03.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8800 IOs, 3593525248 bytes 2024-09-14T05:51:03.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8900 IOs, 3613324288 bytes 2024-09-14T05:51:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9000 IOs, 3663357952 bytes 2024-09-14T05:51:03.516 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9100 IOs, 3736781824 bytes 2024-09-14T05:51:03.626 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9200 IOs, 3749114880 bytes 2024-09-14T05:51:03.761 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9300 IOs, 3805106176 bytes 2024-09-14T05:51:03.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9400 IOs, 3857686528 bytes 2024-09-14T05:51:04.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9500 IOs, 3925510144 bytes 2024-09-14T05:51:04.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9600 IOs, 3977978880 bytes 2024-09-14T05:51:04.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9700 IOs, 4015801344 bytes 2024-09-14T05:51:04.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9800 IOs, 4030891008 bytes 2024-09-14T05:51:04.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9900 IOs, 4080185344 bytes 2024-09-14T05:51:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10000 IOs, 4100529152 bytes 2024-09-14T05:51:04.758 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10100 IOs, 4124459008 bytes 2024-09-14T05:51:04.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:51:04.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:51:04.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:51:04.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:51:04.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:51:04.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_40 2024-09-14T05:51:04.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_40 snapshot 128M 2024-09-14T05:51:04.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:51:04.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:51:04.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:51:04.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:51:04.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_40 2024-09-14T05:51:04.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:51:04.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:51:04.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-14T05:51:04.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:51:04.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_40 128M 2024-09-14T05:51:04.791 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:51:04.791 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:51:04.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:51:04.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:51:04.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_40 2024-09-14T05:51:04.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:51:04.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T05:51:04.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-14T05:51:04.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-14T05:51:04.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:51:04.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_40 2024-09-14T05:51:04.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T05:51:04.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:51:04.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-14T05:51:04.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:51:04.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_40 snapshot 2024-09-14T05:51:04.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:51:04.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:51:04.844 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_40 2024-09-14T05:51:04.844 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:51:04.844 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_40 snapshot 2024-09-14T05:51:05.739 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T05:51:05.751 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_40 2024-09-14T05:51:05.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_40': 2024-09-14T05:51:05.794 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:51:05.794 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:51:05.794 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T05:51:05.794 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 145271b7d180 2024-09-14T05:51:05.794 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.145271b7d180 2024-09-14T05:51:05.794 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:51:05.794 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T05:51:05.794 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:51:05.794 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:51:05.794 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:51:04 2024 2024-09-14T05:51:05.794 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:51:04 2024 2024-09-14T05:51:05.794 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:51:04 2024 2024-09-14T05:51:05.794 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:51:05.795 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:51:05.795 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 5049b9ce-c032-4812-91d0-7dbca1b8b176 2024-09-14T05:51:05.795 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T05:51:05.797 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:51:05.797 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_40 2024-09-14T05:51:05.797 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:51:05.797 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:51:05.797 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_40 2024-09-14T05:51:05.798 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:51:05.799 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=8 2024-09-14T05:51:05.799 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:51:05.799 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_40 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:51:06.018 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 51201024 bytes 2024-09-14T05:51:06.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 118958080 bytes 2024-09-14T05:51:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 147082240 bytes 2024-09-14T05:51:06.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 190688256 bytes 2024-09-14T05:51:06.506 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 197476352 bytes 2024-09-14T05:51:06.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 254235648 bytes 2024-09-14T05:51:06.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 308889600 bytes 2024-09-14T05:51:06.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 346956800 bytes 2024-09-14T05:51:07.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 391515136 bytes 2024-09-14T05:51:07.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 442923008 bytes 2024-09-14T05:51:07.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 443551744 bytes 2024-09-14T05:51:07.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 496384000 bytes 2024-09-14T05:51:07.548 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 558340096 bytes 2024-09-14T05:51:07.636 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 581196800 bytes 2024-09-14T05:51:07.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 598191104 bytes 2024-09-14T05:51:07.889 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 641030144 bytes 2024-09-14T05:51:07.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 676488192 bytes 2024-09-14T05:51:08.117 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 705581056 bytes 2024-09-14T05:51:08.262 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 758800384 bytes 2024-09-14T05:51:08.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 825016320 bytes 2024-09-14T05:51:08.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 869678080 bytes 2024-09-14T05:51:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 939894784 bytes 2024-09-14T05:51:08.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1005887488 bytes 2024-09-14T05:51:08.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1027462144 bytes 2024-09-14T05:51:09.069 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1038706688 bytes 2024-09-14T05:51:09.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1090883584 bytes 2024-09-14T05:51:09.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1155142656 bytes 2024-09-14T05:51:09.489 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1213641728 bytes 2024-09-14T05:51:09.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1267162112 bytes 2024-09-14T05:51:09.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1352592384 bytes 2024-09-14T05:51:09.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1416646656 bytes 2024-09-14T05:51:10.073 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1441771520 bytes 2024-09-14T05:51:10.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1487973376 bytes 2024-09-14T05:51:10.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1566844928 bytes 2024-09-14T05:51:10.532 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1645021184 bytes 2024-09-14T05:51:10.646 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1668385792 bytes 2024-09-14T05:51:10.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1721172992 bytes 2024-09-14T05:51:10.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1740591104 bytes 2024-09-14T05:51:10.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1753824256 bytes 2024-09-14T05:51:11.119 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1806895104 bytes 2024-09-14T05:51:11.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1815254016 bytes 2024-09-14T05:51:11.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1842845696 bytes 2024-09-14T05:51:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1881009152 bytes 2024-09-14T05:51:11.548 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1888677888 bytes 2024-09-14T05:51:11.639 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1897117696 bytes 2024-09-14T05:51:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1918099456 bytes 2024-09-14T05:51:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 1986834432 bytes 2024-09-14T05:51:12.022 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2016774144 bytes 2024-09-14T05:51:12.128 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2041457664 bytes 2024-09-14T05:51:12.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2112015360 bytes 2024-09-14T05:51:12.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2183259136 bytes 2024-09-14T05:51:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2222812160 bytes 2024-09-14T05:51:12.710 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2288277504 bytes 2024-09-14T05:51:12.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2312334336 bytes 2024-09-14T05:51:12.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2363877376 bytes 2024-09-14T05:51:13.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2388634624 bytes 2024-09-14T05:51:13.220 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2461012992 bytes 2024-09-14T05:51:13.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2516202496 bytes 2024-09-14T05:51:13.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2549558272 bytes 2024-09-14T05:51:13.578 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2563776512 bytes 2024-09-14T05:51:13.721 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2628860928 bytes 2024-09-14T05:51:13.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:51:13.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:51:13.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:51:13.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:51:13.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:51:13.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_41 2024-09-14T05:51:13.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_41 snapshot 128M 2024-09-14T05:51:13.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:51:13.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:51:13.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:51:13.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:51:13.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_41 2024-09-14T05:51:13.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:51:13.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:51:13.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-14T05:51:13.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:51:13.808 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_41 128M 2024-09-14T05:51:13.808 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:51:13.808 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:51:13.808 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:51:13.808 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:51:13.808 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_41 2024-09-14T05:51:13.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:51:13.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T05:51:13.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-14T05:51:13.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-14T05:51:13.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:51:13.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_41 2024-09-14T05:51:13.858 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T05:51:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:51:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-14T05:51:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:51:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_41 snapshot 2024-09-14T05:51:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:51:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:51:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_41 2024-09-14T05:51:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:51:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_41 snapshot 2024-09-14T05:51:14.756 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T05:51:14.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_41 2024-09-14T05:51:14.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_41': 2024-09-14T05:51:14.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:51:14.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:51:14.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T05:51:14.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 145e894b77ae 2024-09-14T05:51:14.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.145e894b77ae 2024-09-14T05:51:14.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:51:14.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T05:51:14.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:51:14.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:51:14.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:51:13 2024 2024-09-14T05:51:14.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:51:13 2024 2024-09-14T05:51:14.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:51:13 2024 2024-09-14T05:51:14.811 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:51:14.811 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:51:14.811 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 20523f16-1a82-463b-833f-79e70faaf271 2024-09-14T05:51:14.811 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T05:51:14.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:51:14.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_41 2024-09-14T05:51:14.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:51:14.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:51:14.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_41 2024-09-14T05:51:14.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:51:14.815 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=9 2024-09-14T05:51:14.815 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:51:14.815 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_41 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:51:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 20002816 bytes 2024-09-14T05:51:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 87469056 bytes 2024-09-14T05:51:15.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 117823488 bytes 2024-09-14T05:51:15.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 171253760 bytes 2024-09-14T05:51:15.542 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 225609728 bytes 2024-09-14T05:51:15.699 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 305469440 bytes 2024-09-14T05:51:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 334558208 bytes 2024-09-14T05:51:15.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 376344576 bytes 2024-09-14T05:51:16.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 416657408 bytes 2024-09-14T05:51:16.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 430131200 bytes 2024-09-14T05:51:16.301 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 479854592 bytes 2024-09-14T05:51:16.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 589886464 bytes 2024-09-14T05:51:16.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 641507328 bytes 2024-09-14T05:51:16.737 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 658972672 bytes 2024-09-14T05:51:16.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 670607360 bytes 2024-09-14T05:51:16.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 714187776 bytes 2024-09-14T05:51:17.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 734661632 bytes 2024-09-14T05:51:17.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 782085120 bytes 2024-09-14T05:51:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 834061312 bytes 2024-09-14T05:51:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 870256640 bytes 2024-09-14T05:51:17.529 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 870881280 bytes 2024-09-14T05:51:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 891601920 bytes 2024-09-14T05:51:17.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 963499008 bytes 2024-09-14T05:51:17.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 988959744 bytes 2024-09-14T05:51:18.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1007248384 bytes 2024-09-14T05:51:18.119 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1040497664 bytes 2024-09-14T05:51:18.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1050857472 bytes 2024-09-14T05:51:18.343 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1091434496 bytes 2024-09-14T05:51:18.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1106174976 bytes 2024-09-14T05:51:18.549 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1143160832 bytes 2024-09-14T05:51:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1161040896 bytes 2024-09-14T05:51:18.751 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1164414976 bytes 2024-09-14T05:51:18.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1220534272 bytes 2024-09-14T05:51:19.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1274645504 bytes 2024-09-14T05:51:19.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1291502592 bytes 2024-09-14T05:51:19.271 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1349304320 bytes 2024-09-14T05:51:19.387 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1382296576 bytes 2024-09-14T05:51:19.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1416084480 bytes 2024-09-14T05:51:19.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1431876608 bytes 2024-09-14T05:51:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1485107200 bytes 2024-09-14T05:51:19.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1523407872 bytes 2024-09-14T05:51:19.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1566678016 bytes 2024-09-14T05:51:20.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1630263296 bytes 2024-09-14T05:51:20.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1643912192 bytes 2024-09-14T05:51:20.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1680578560 bytes 2024-09-14T05:51:20.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1731529728 bytes 2024-09-14T05:51:20.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 1793390592 bytes 2024-09-14T05:51:20.749 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 1823349760 bytes 2024-09-14T05:51:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 1866577920 bytes 2024-09-14T05:51:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 1896636416 bytes 2024-09-14T05:51:21.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 1946838016 bytes 2024-09-14T05:51:21.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 1995262976 bytes 2024-09-14T05:51:21.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2061436928 bytes 2024-09-14T05:51:21.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2137617408 bytes 2024-09-14T05:51:21.688 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2158339072 bytes 2024-09-14T05:51:21.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2208622592 bytes 2024-09-14T05:51:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2243638272 bytes 2024-09-14T05:51:22.038 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2249709568 bytes 2024-09-14T05:51:22.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2349615104 bytes 2024-09-14T05:51:22.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2399150080 bytes 2024-09-14T05:51:22.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2445956096 bytes 2024-09-14T05:51:22.599 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2467927040 bytes 2024-09-14T05:51:22.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2539048960 bytes 2024-09-14T05:51:22.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2599120896 bytes 2024-09-14T05:51:23.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2614790144 bytes 2024-09-14T05:51:23.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 2654006272 bytes 2024-09-14T05:51:23.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 2681510912 bytes 2024-09-14T05:51:23.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 2705601536 bytes 2024-09-14T05:51:23.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 2751386624 bytes 2024-09-14T05:51:23.639 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 2830141440 bytes 2024-09-14T05:51:23.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 2830752768 bytes 2024-09-14T05:51:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:51:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:51:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:51:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:51:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:51:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_42 2024-09-14T05:51:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_42 snapshot 128M 2024-09-14T05:51:23.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:51:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:51:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:51:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:51:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_42 2024-09-14T05:51:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:51:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:51:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-14T05:51:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:51:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_42 128M 2024-09-14T05:51:23.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:51:23.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:51:23.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:51:23.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:51:23.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_42 2024-09-14T05:51:23.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:51:23.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T05:51:23.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-14T05:51:23.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-14T05:51:23.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:51:23.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_42 2024-09-14T05:51:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T05:51:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:51:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-14T05:51:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:51:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_42 snapshot 2024-09-14T05:51:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:51:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:51:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_42 2024-09-14T05:51:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:51:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_42 snapshot 2024-09-14T05:51:24.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T05:51:24.787 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_42 2024-09-14T05:51:24.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_42': 2024-09-14T05:51:24.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:51:24.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:51:24.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T05:51:24.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 146a42b69578 2024-09-14T05:51:24.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.146a42b69578 2024-09-14T05:51:24.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:51:24.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T05:51:24.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:51:24.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:51:24.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:51:23 2024 2024-09-14T05:51:24.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:51:23 2024 2024-09-14T05:51:24.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:51:23 2024 2024-09-14T05:51:24.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:51:24.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:51:24.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 662fa768-1dea-4f15-91bf-edf297ebd84a 2024-09-14T05:51:24.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T05:51:24.831 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:51:24.831 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_42 2024-09-14T05:51:24.831 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:51:24.831 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:51:24.831 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_42 2024-09-14T05:51:24.831 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:51:24.832 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=9 2024-09-14T05:51:24.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:51:24.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_42 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:51:25.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 60644352 bytes 2024-09-14T05:51:25.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 81993728 bytes 2024-09-14T05:51:25.273 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 105750528 bytes 2024-09-14T05:51:25.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 149954560 bytes 2024-09-14T05:51:25.559 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 223660032 bytes 2024-09-14T05:51:25.667 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 256586752 bytes 2024-09-14T05:51:25.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 295478272 bytes 2024-09-14T05:51:25.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 360325120 bytes 2024-09-14T05:51:26.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 420536320 bytes 2024-09-14T05:51:26.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 453504000 bytes 2024-09-14T05:51:26.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 577507328 bytes 2024-09-14T05:51:26.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 603346944 bytes 2024-09-14T05:51:26.638 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 626395136 bytes 2024-09-14T05:51:26.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 627002368 bytes 2024-09-14T05:51:26.855 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 663514112 bytes 2024-09-14T05:51:26.952 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 676352000 bytes 2024-09-14T05:51:27.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 741467136 bytes 2024-09-14T05:51:27.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 795070464 bytes 2024-09-14T05:51:27.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 811078656 bytes 2024-09-14T05:51:27.456 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 833524736 bytes 2024-09-14T05:51:27.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 861442048 bytes 2024-09-14T05:51:27.708 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 908510208 bytes 2024-09-14T05:51:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 939950080 bytes 2024-09-14T05:51:27.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 985410560 bytes 2024-09-14T05:51:28.096 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1041912832 bytes 2024-09-14T05:51:28.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1094542336 bytes 2024-09-14T05:51:28.437 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1161453568 bytes 2024-09-14T05:51:28.552 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1182168064 bytes 2024-09-14T05:51:28.659 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1225494528 bytes 2024-09-14T05:51:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1293544448 bytes 2024-09-14T05:51:28.969 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1337072640 bytes 2024-09-14T05:51:29.102 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1384532992 bytes 2024-09-14T05:51:29.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1424544768 bytes 2024-09-14T05:51:29.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1465538560 bytes 2024-09-14T05:51:29.502 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1522480128 bytes 2024-09-14T05:51:29.626 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1555996672 bytes 2024-09-14T05:51:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1556619264 bytes 2024-09-14T05:51:29.821 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1581053952 bytes 2024-09-14T05:51:29.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1651805184 bytes 2024-09-14T05:51:30.076 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1667752960 bytes 2024-09-14T05:51:30.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1711067136 bytes 2024-09-14T05:51:30.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1735359488 bytes 2024-09-14T05:51:30.436 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1765488640 bytes 2024-09-14T05:51:30.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1817944064 bytes 2024-09-14T05:51:30.738 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1890066432 bytes 2024-09-14T05:51:30.844 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1910334464 bytes 2024-09-14T05:51:30.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 1945458688 bytes 2024-09-14T05:51:31.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 1989703680 bytes 2024-09-14T05:51:31.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2001629184 bytes 2024-09-14T05:51:31.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2030600192 bytes 2024-09-14T05:51:31.406 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2041900032 bytes 2024-09-14T05:51:31.557 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2123866112 bytes 2024-09-14T05:51:31.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2192544768 bytes 2024-09-14T05:51:31.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2232579072 bytes 2024-09-14T05:51:32.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2321958912 bytes 2024-09-14T05:51:32.228 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2433103872 bytes 2024-09-14T05:51:32.381 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2482716672 bytes 2024-09-14T05:51:32.542 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2551732224 bytes 2024-09-14T05:51:32.699 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2637661184 bytes 2024-09-14T05:51:32.856 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2698319872 bytes 2024-09-14T05:51:32.868 INFO:tasks.ceph.cluster2.osd.1.smithi191.stderr:2024-09-14T05:51:32.867+0000 7f9375a93640 -1 reset not still connected to 0x5611e62628f0 2024-09-14T05:51:32.869 INFO:tasks.ceph.cluster2.osd.1.smithi191.stderr:2024-09-14T05:51:32.867+0000 7f9375a93640 -1 reset not still connected to 0x5611e6ae7790 2024-09-14T05:51:32.869 INFO:tasks.ceph.cluster2.osd.1.smithi191.stderr:2024-09-14T05:51:32.867+0000 7f9375a93640 -1 reset not still connected to 0x5611e70609c0 2024-09-14T05:51:32.870 INFO:tasks.ceph.cluster2.osd.1.smithi191.stderr:2024-09-14T05:51:32.869+0000 7f9375a93640 -1 reset not still connected to 0x5611e7cd3d40 2024-09-14T05:51:32.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2738793472 bytes 2024-09-14T05:51:33.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2775861248 bytes 2024-09-14T05:51:33.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2791794688 bytes 2024-09-14T05:51:33.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2821164032 bytes 2024-09-14T05:51:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2832389120 bytes 2024-09-14T05:51:33.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 2834930688 bytes 2024-09-14T05:51:33.761 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 2925830144 bytes 2024-09-14T05:51:33.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:51:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:51:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:51:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:51:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:51:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_43 2024-09-14T05:51:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_43 snapshot 128M 2024-09-14T05:51:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:51:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:51:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:51:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:51:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_43 2024-09-14T05:51:33.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:51:33.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:51:33.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-14T05:51:33.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:51:33.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_43 128M 2024-09-14T05:51:33.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:51:33.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:51:33.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:51:33.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:51:33.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_43 2024-09-14T05:51:33.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:51:33.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T05:51:33.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-14T05:51:33.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-14T05:51:33.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:51:33.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_43 2024-09-14T05:51:34.097 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T05:51:34.097 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:51:34.097 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-14T05:51:34.097 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:51:34.097 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_43 snapshot 2024-09-14T05:51:34.097 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:51:34.097 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:51:34.097 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_43 2024-09-14T05:51:34.097 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:51:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_43 snapshot 2024-09-14T05:51:34.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T05:51:34.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_43 2024-09-14T05:51:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_43': 2024-09-14T05:51:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:51:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:51:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T05:51:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 14765da162ee 2024-09-14T05:51:34.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.14765da162ee 2024-09-14T05:51:34.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:51:34.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T05:51:34.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:51:34.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:51:34.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:51:34 2024 2024-09-14T05:51:34.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:51:34 2024 2024-09-14T05:51:34.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:51:34 2024 2024-09-14T05:51:34.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:51:34.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:51:34.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: f71c9e7c-cbd0-4274-bae7-2c98cc5052b1 2024-09-14T05:51:34.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T05:51:34.863 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:51:34.863 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_43 2024-09-14T05:51:34.864 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:51:34.864 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:51:34.864 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_43 2024-09-14T05:51:34.864 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:51:34.865 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=9 2024-09-14T05:51:34.865 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:51:34.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_43 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:51:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 41278464 bytes 2024-09-14T05:51:35.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 67236864 bytes 2024-09-14T05:51:35.295 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 84985856 bytes 2024-09-14T05:51:35.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 111313920 bytes 2024-09-14T05:51:35.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 143989760 bytes 2024-09-14T05:51:35.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 185604096 bytes 2024-09-14T05:51:35.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 233572352 bytes 2024-09-14T05:51:35.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 323760128 bytes 2024-09-14T05:51:36.080 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 362361856 bytes 2024-09-14T05:51:36.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 413298688 bytes 2024-09-14T05:51:36.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 443994112 bytes 2024-09-14T05:51:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 484683776 bytes 2024-09-14T05:51:36.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 513343488 bytes 2024-09-14T05:51:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 568518656 bytes 2024-09-14T05:51:36.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 590734336 bytes 2024-09-14T05:51:36.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 639349760 bytes 2024-09-14T05:51:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 674076672 bytes 2024-09-14T05:51:37.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 714769408 bytes 2024-09-14T05:51:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 799792128 bytes 2024-09-14T05:51:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 824812544 bytes 2024-09-14T05:51:37.615 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 875037696 bytes 2024-09-14T05:51:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 875638784 bytes 2024-09-14T05:51:37.856 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 939753472 bytes 2024-09-14T05:51:37.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 943600640 bytes 2024-09-14T05:51:38.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1012300800 bytes 2024-09-14T05:51:38.193 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1016422400 bytes 2024-09-14T05:51:38.295 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1035812864 bytes 2024-09-14T05:51:38.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1069584384 bytes 2024-09-14T05:51:38.549 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1132765184 bytes 2024-09-14T05:51:38.708 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1182099456 bytes 2024-09-14T05:51:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1216707584 bytes 2024-09-14T05:51:38.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1278210048 bytes 2024-09-14T05:51:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1321421824 bytes 2024-09-14T05:51:39.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1361925120 bytes 2024-09-14T05:51:39.368 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1403571200 bytes 2024-09-14T05:51:39.473 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1426088960 bytes 2024-09-14T05:51:39.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1485684736 bytes 2024-09-14T05:51:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1526777856 bytes 2024-09-14T05:51:39.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1527398400 bytes 2024-09-14T05:51:39.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1575710720 bytes 2024-09-14T05:51:40.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1589195776 bytes 2024-09-14T05:51:40.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1613340672 bytes 2024-09-14T05:51:40.287 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1638536192 bytes 2024-09-14T05:51:40.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1660863488 bytes 2024-09-14T05:51:40.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1683196928 bytes 2024-09-14T05:51:40.619 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1696833536 bytes 2024-09-14T05:51:40.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 1772110848 bytes 2024-09-14T05:51:40.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 1846707200 bytes 2024-09-14T05:51:41.080 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 1886883840 bytes 2024-09-14T05:51:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 1934925824 bytes 2024-09-14T05:51:41.329 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 1960791040 bytes 2024-09-14T05:51:41.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2034799616 bytes 2024-09-14T05:51:41.654 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2113579008 bytes 2024-09-14T05:51:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2196229120 bytes 2024-09-14T05:51:41.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2223283200 bytes 2024-09-14T05:51:42.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2278822912 bytes 2024-09-14T05:51:42.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2296157184 bytes 2024-09-14T05:51:42.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2319209472 bytes 2024-09-14T05:51:42.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2369216512 bytes 2024-09-14T05:51:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2432442368 bytes 2024-09-14T05:51:42.669 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2469444608 bytes 2024-09-14T05:51:42.830 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2513895424 bytes 2024-09-14T05:51:42.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2516934656 bytes 2024-09-14T05:51:43.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2565397504 bytes 2024-09-14T05:51:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2641107968 bytes 2024-09-14T05:51:43.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 2687365120 bytes 2024-09-14T05:51:43.497 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8333306520966024) > weight (0.25) 2024-09-14T05:51:43.497 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.36632313111139037) > weight (0.25) 2024-09-14T05:51:43.497 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.8418201260387933) > weight (0.25) 2024-09-14T05:51:43.497 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.586556623968624) > weight (0.25) 2024-09-14T05:51:43.498 INFO:tasks.rbd_mirror_thrash:waiting for 75 secs before thrashing 2024-09-14T05:51:43.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 2718380032 bytes 2024-09-14T05:51:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 2719013888 bytes 2024-09-14T05:51:43.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 2740251648 bytes 2024-09-14T05:51:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 2766098432 bytes 2024-09-14T05:51:43.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:51:43.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:51:43.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:51:43.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:51:43.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:51:43.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_44 2024-09-14T05:51:43.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_44 snapshot 128M 2024-09-14T05:51:43.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:51:43.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:51:43.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:51:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:51:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_44 2024-09-14T05:51:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:51:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:51:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-14T05:51:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:51:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_44 128M 2024-09-14T05:51:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:51:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:51:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:51:43.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:51:43.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_44 2024-09-14T05:51:43.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:51:43.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T05:51:43.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-14T05:51:43.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-14T05:51:43.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:51:43.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_44 2024-09-14T05:51:43.931 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T05:51:43.931 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:51:43.932 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-14T05:51:43.932 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:51:43.932 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_44 snapshot 2024-09-14T05:51:43.932 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:51:43.932 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:51:43.932 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_44 2024-09-14T05:51:43.932 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:51:43.932 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_44 snapshot 2024-09-14T05:51:44.851 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T05:51:44.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_44 2024-09-14T05:51:44.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_44': 2024-09-14T05:51:44.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:51:44.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:51:44.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T05:51:44.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 1482ef469252 2024-09-14T05:51:44.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.1482ef469252 2024-09-14T05:51:44.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:51:44.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T05:51:44.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:51:44.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:51:44.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:51:43 2024 2024-09-14T05:51:44.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:51:43 2024 2024-09-14T05:51:44.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:51:43 2024 2024-09-14T05:51:44.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:51:44.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:51:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 40153520-d46c-4ade-9ebf-442136d19014 2024-09-14T05:51:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T05:51:44.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:51:44.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_44 2024-09-14T05:51:44.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:51:44.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:51:44.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_44 2024-09-14T05:51:44.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:51:44.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=10 2024-09-14T05:51:44.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:51:44.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_44 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:51:45.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 30139392 bytes 2024-09-14T05:51:45.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 99431424 bytes 2024-09-14T05:51:45.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 178026496 bytes 2024-09-14T05:51:45.562 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 223791104 bytes 2024-09-14T05:51:45.734 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 314098688 bytes 2024-09-14T05:51:45.855 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 348923904 bytes 2024-09-14T05:51:46.018 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 423572480 bytes 2024-09-14T05:51:46.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 440591360 bytes 2024-09-14T05:51:46.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 488401920 bytes 2024-09-14T05:51:46.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 494629888 bytes 2024-09-14T05:51:46.478 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 544566272 bytes 2024-09-14T05:51:46.623 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 604651520 bytes 2024-09-14T05:51:46.734 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 634668032 bytes 2024-09-14T05:51:46.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 660123648 bytes 2024-09-14T05:51:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 721475584 bytes 2024-09-14T05:51:47.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 734595072 bytes 2024-09-14T05:51:47.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 789328896 bytes 2024-09-14T05:51:47.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 804655104 bytes 2024-09-14T05:51:47.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 873477120 bytes 2024-09-14T05:51:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 919044096 bytes 2024-09-14T05:51:47.731 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 927783936 bytes 2024-09-14T05:51:47.822 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 945290240 bytes 2024-09-14T05:51:47.952 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 971463680 bytes 2024-09-14T05:51:48.100 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1030310912 bytes 2024-09-14T05:51:48.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1066777600 bytes 2024-09-14T05:51:48.375 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1143072768 bytes 2024-09-14T05:51:48.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1186440192 bytes 2024-09-14T05:51:48.600 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1200131072 bytes 2024-09-14T05:51:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1251765248 bytes 2024-09-14T05:51:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1320928256 bytes 2024-09-14T05:51:49.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1345210368 bytes 2024-09-14T05:51:49.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1400223744 bytes 2024-09-14T05:51:49.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1424699392 bytes 2024-09-14T05:51:49.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1498587136 bytes 2024-09-14T05:51:49.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1543403520 bytes 2024-09-14T05:51:49.684 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1586506752 bytes 2024-09-14T05:51:49.808 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1624339456 bytes 2024-09-14T05:51:49.951 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1680316416 bytes 2024-09-14T05:51:50.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1711232000 bytes 2024-09-14T05:51:50.193 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1743363072 bytes 2024-09-14T05:51:50.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1798523904 bytes 2024-09-14T05:51:50.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1908672512 bytes 2024-09-14T05:51:50.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1964617728 bytes 2024-09-14T05:51:50.791 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1990301696 bytes 2024-09-14T05:51:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 2045607936 bytes 2024-09-14T05:51:51.076 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 2102180864 bytes 2024-09-14T05:51:51.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 2165855232 bytes 2024-09-14T05:51:51.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2207096832 bytes 2024-09-14T05:51:51.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2278477824 bytes 2024-09-14T05:51:51.600 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2281873408 bytes 2024-09-14T05:51:51.732 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2325182464 bytes 2024-09-14T05:51:51.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2418582528 bytes 2024-09-14T05:51:52.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2445500416 bytes 2024-09-14T05:51:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2516700160 bytes 2024-09-14T05:51:52.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2530449408 bytes 2024-09-14T05:51:52.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2545881088 bytes 2024-09-14T05:51:52.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2609957888 bytes 2024-09-14T05:51:52.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2610588672 bytes 2024-09-14T05:51:52.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2659452928 bytes 2024-09-14T05:51:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2736710656 bytes 2024-09-14T05:51:53.040 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2764284928 bytes 2024-09-14T05:51:53.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2816799744 bytes 2024-09-14T05:51:53.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2869258240 bytes 2024-09-14T05:51:53.422 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2918796288 bytes 2024-09-14T05:51:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2976562176 bytes 2024-09-14T05:51:53.703 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 3016902656 bytes 2024-09-14T05:51:54.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 3070736384 bytes 2024-09-14T05:51:54.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 3134355456 bytes 2024-09-14T05:51:54.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 3187487744 bytes 2024-09-14T05:51:54.457 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 3206328320 bytes 2024-09-14T05:51:54.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 3250837504 bytes 2024-09-14T05:51:54.694 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 3272821760 bytes 2024-09-14T05:51:54.815 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7300 IOs, 3301943296 bytes 2024-09-14T05:51:54.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:51:54.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:51:54.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:51:54.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:51:54.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:51:54.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_45 2024-09-14T05:51:54.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_45 snapshot 128M 2024-09-14T05:51:54.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:51:54.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:51:54.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:51:54.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:51:54.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_45 2024-09-14T05:51:54.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:51:54.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:51:54.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-14T05:51:54.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:51:54.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_45 128M 2024-09-14T05:51:54.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:51:54.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:51:54.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:51:54.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:51:54.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_45 2024-09-14T05:51:54.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:51:54.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T05:51:54.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-14T05:51:54.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-14T05:51:54.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:51:54.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_45 2024-09-14T05:51:54.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T05:51:54.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:51:54.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-14T05:51:54.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:51:54.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_45 snapshot 2024-09-14T05:51:54.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:51:54.969 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:51:54.969 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_45 2024-09-14T05:51:54.969 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:51:54.969 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_45 snapshot 2024-09-14T05:51:55.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T05:51:55.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_45 2024-09-14T05:51:55.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_45': 2024-09-14T05:51:55.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:51:55.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:51:55.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T05:51:55.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 148e94338c1e 2024-09-14T05:51:55.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.148e94338c1e 2024-09-14T05:51:55.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:51:55.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T05:51:55.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:51:55.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:51:55.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:51:54 2024 2024-09-14T05:51:55.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:51:54 2024 2024-09-14T05:51:55.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:51:54 2024 2024-09-14T05:51:55.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:51:55.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:51:55.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 8ed3596b-4b0c-421c-ae91-4a8f4c37bca1 2024-09-14T05:51:55.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T05:51:55.926 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:51:55.926 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_45 2024-09-14T05:51:55.926 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:51:55.926 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:51:55.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_45 2024-09-14T05:51:55.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:51:55.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=10 2024-09-14T05:51:55.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:51:55.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_45 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:51:56.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 53729280 bytes 2024-09-14T05:51:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 54342656 bytes 2024-09-14T05:51:56.341 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 75074560 bytes 2024-09-14T05:51:56.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 181173248 bytes 2024-09-14T05:51:56.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 213961728 bytes 2024-09-14T05:51:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 321120256 bytes 2024-09-14T05:51:56.978 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 374160384 bytes 2024-09-14T05:51:57.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 435544064 bytes 2024-09-14T05:51:57.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 507968512 bytes 2024-09-14T05:51:57.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 583705600 bytes 2024-09-14T05:51:57.570 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 634142720 bytes 2024-09-14T05:51:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 671297536 bytes 2024-09-14T05:51:57.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 734075904 bytes 2024-09-14T05:51:57.960 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 763965440 bytes 2024-09-14T05:51:58.062 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 780250112 bytes 2024-09-14T05:51:58.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 839826432 bytes 2024-09-14T05:51:58.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 862301184 bytes 2024-09-14T05:51:58.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 895328256 bytes 2024-09-14T05:51:58.516 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 895956992 bytes 2024-09-14T05:51:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 925088768 bytes 2024-09-14T05:51:58.756 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 961263616 bytes 2024-09-14T05:51:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 972524544 bytes 2024-09-14T05:51:58.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1016588288 bytes 2024-09-14T05:51:59.069 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1023538176 bytes 2024-09-14T05:51:59.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1101417472 bytes 2024-09-14T05:51:59.341 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1134283776 bytes 2024-09-14T05:51:59.437 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1137533952 bytes 2024-09-14T05:51:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1214619648 bytes 2024-09-14T05:51:59.738 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1266269184 bytes 2024-09-14T05:51:59.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1310522368 bytes 2024-09-14T05:51:59.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1311144960 bytes 2024-09-14T05:52:00.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1344189440 bytes 2024-09-14T05:52:00.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1355804672 bytes 2024-09-14T05:52:00.341 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1448027136 bytes 2024-09-14T05:52:00.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1507598336 bytes 2024-09-14T05:52:00.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1568351232 bytes 2024-09-14T05:52:00.776 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1625229312 bytes 2024-09-14T05:52:00.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1658530816 bytes 2024-09-14T05:52:00.996 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1673231360 bytes 2024-09-14T05:52:01.096 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1694869504 bytes 2024-09-14T05:52:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1759856640 bytes 2024-09-14T05:52:01.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1775480832 bytes 2024-09-14T05:52:01.478 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1818120192 bytes 2024-09-14T05:52:01.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1866721280 bytes 2024-09-14T05:52:01.708 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1878785024 bytes 2024-09-14T05:52:01.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1903246336 bytes 2024-09-14T05:52:01.952 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 1961937920 bytes 2024-09-14T05:52:02.073 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 1982121984 bytes 2024-09-14T05:52:02.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2010147840 bytes 2024-09-14T05:52:02.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2022159360 bytes 2024-09-14T05:52:02.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2042865664 bytes 2024-09-14T05:52:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2085587968 bytes 2024-09-14T05:52:02.600 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2093260800 bytes 2024-09-14T05:52:02.708 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2119851008 bytes 2024-09-14T05:52:02.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2202444800 bytes 2024-09-14T05:52:02.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2235432960 bytes 2024-09-14T05:52:03.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2278807552 bytes 2024-09-14T05:52:03.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2326590464 bytes 2024-09-14T05:52:03.387 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2377204736 bytes 2024-09-14T05:52:03.600 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2491544576 bytes 2024-09-14T05:52:03.734 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2537775104 bytes 2024-09-14T05:52:03.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2573712384 bytes 2024-09-14T05:52:04.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2648458240 bytes 2024-09-14T05:52:04.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2670327808 bytes 2024-09-14T05:52:04.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2695435264 bytes 2024-09-14T05:52:04.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 2725786624 bytes 2024-09-14T05:52:04.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 2745975808 bytes 2024-09-14T05:52:04.596 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 2790123520 bytes 2024-09-14T05:52:04.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 2824354816 bytes 2024-09-14T05:52:04.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 2872843264 bytes 2024-09-14T05:52:04.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 2914382848 bytes 2024-09-14T05:52:05.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 2966603776 bytes 2024-09-14T05:52:05.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7300 IOs, 3013567488 bytes 2024-09-14T05:52:05.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7400 IOs, 3065488384 bytes 2024-09-14T05:52:05.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7500 IOs, 3117824000 bytes 2024-09-14T05:52:05.655 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7600 IOs, 3161559040 bytes 2024-09-14T05:52:05.797 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7700 IOs, 3221694464 bytes 2024-09-14T05:52:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:52:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:52:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:52:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:52:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:52:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_46 2024-09-14T05:52:05.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_46 snapshot 128M 2024-09-14T05:52:05.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:52:05.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:52:05.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:52:05.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:52:05.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_46 2024-09-14T05:52:05.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:52:05.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:52:05.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-14T05:52:05.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:52:05.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_46 128M 2024-09-14T05:52:05.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:52:05.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:52:05.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:52:05.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:52:05.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_46 2024-09-14T05:52:05.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:52:05.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T05:52:05.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-14T05:52:05.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-14T05:52:05.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:52:05.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_46 2024-09-14T05:52:05.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T05:52:05.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:52:05.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-14T05:52:05.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:52:05.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_46 snapshot 2024-09-14T05:52:05.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:52:05.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:52:05.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_46 2024-09-14T05:52:05.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:52:05.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_46 snapshot 2024-09-14T05:52:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T05:52:06.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_46 2024-09-14T05:52:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_46': 2024-09-14T05:52:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:52:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:52:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T05:52:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 149a585921c6 2024-09-14T05:52:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.149a585921c6 2024-09-14T05:52:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:52:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T05:52:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:52:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:52:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:52:05 2024 2024-09-14T05:52:06.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:52:05 2024 2024-09-14T05:52:06.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:52:05 2024 2024-09-14T05:52:06.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:52:06.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:52:06.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 53467864-fd16-454f-8435-46f986b18a2f 2024-09-14T05:52:06.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T05:52:06.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:52:06.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_46 2024-09-14T05:52:06.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:52:06.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:52:06.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_46 2024-09-14T05:52:06.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:52:06.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=14 2024-09-14T05:52:06.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:52:06.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_46 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:52:07.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 50759680 bytes 2024-09-14T05:52:07.312 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 116864000 bytes 2024-09-14T05:52:07.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 166260736 bytes 2024-09-14T05:52:07.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 206183424 bytes 2024-09-14T05:52:07.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 233360384 bytes 2024-09-14T05:52:07.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 280946688 bytes 2024-09-14T05:52:07.911 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 296102912 bytes 2024-09-14T05:52:08.055 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 353828864 bytes 2024-09-14T05:52:08.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 470584320 bytes 2024-09-14T05:52:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 533090304 bytes 2024-09-14T05:52:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 638779392 bytes 2024-09-14T05:52:08.728 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 666294272 bytes 2024-09-14T05:52:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 687084544 bytes 2024-09-14T05:52:08.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 694190080 bytes 2024-09-14T05:52:09.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 748815360 bytes 2024-09-14T05:52:09.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 788712448 bytes 2024-09-14T05:52:09.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 882092032 bytes 2024-09-14T05:52:09.499 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 922500096 bytes 2024-09-14T05:52:09.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 937593856 bytes 2024-09-14T05:52:09.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 984126464 bytes 2024-09-14T05:52:09.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 1008496640 bytes 2024-09-14T05:52:10.012 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 1092269056 bytes 2024-09-14T05:52:10.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1153969152 bytes 2024-09-14T05:52:10.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1220324352 bytes 2024-09-14T05:52:10.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1288130560 bytes 2024-09-14T05:52:10.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1355153408 bytes 2024-09-14T05:52:10.733 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1384594432 bytes 2024-09-14T05:52:10.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1445607424 bytes 2024-09-14T05:52:11.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1507036160 bytes 2024-09-14T05:52:11.171 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1558347776 bytes 2024-09-14T05:52:11.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1608437760 bytes 2024-09-14T05:52:11.458 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1672905728 bytes 2024-09-14T05:52:11.481 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6726315605214888) > weight (0.25) 2024-09-14T05:52:11.481 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-09-14T05:52:11.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-09-14T05:52:11.482 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-09-14T05:52:11.483 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:2024-09-14T05:52:11.482+0000 7f6bf4417640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 48497) UID: 1000 2024-09-14T05:52:11.612 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1734526976 bytes 2024-09-14T05:52:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1803829248 bytes 2024-09-14T05:52:11.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1849974784 bytes 2024-09-14T05:52:12.012 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1879305216 bytes 2024-09-14T05:52:12.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1924707328 bytes 2024-09-14T05:52:12.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1959371776 bytes 2024-09-14T05:52:12.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1992612864 bytes 2024-09-14T05:52:12.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 2031014912 bytes 2024-09-14T05:52:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 2067478528 bytes 2024-09-14T05:52:12.728 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 2079383552 bytes 2024-09-14T05:52:12.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 2102434816 bytes 2024-09-14T05:52:12.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 2159875072 bytes 2024-09-14T05:52:13.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 2233149440 bytes 2024-09-14T05:52:13.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 2237881344 bytes 2024-09-14T05:52:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 2317889536 bytes 2024-09-14T05:52:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2322048000 bytes 2024-09-14T05:52:13.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2358225920 bytes 2024-09-14T05:52:13.778 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2432349184 bytes 2024-09-14T05:52:13.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2474571776 bytes 2024-09-14T05:52:14.040 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2536511488 bytes 2024-09-14T05:52:14.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2584843264 bytes 2024-09-14T05:52:14.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2636986368 bytes 2024-09-14T05:52:14.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2668962816 bytes 2024-09-14T05:52:14.545 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2690088960 bytes 2024-09-14T05:52:14.687 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2747163648 bytes 2024-09-14T05:52:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2860917760 bytes 2024-09-14T05:52:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2889775104 bytes 2024-09-14T05:52:15.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2890394624 bytes 2024-09-14T05:52:15.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2928934912 bytes 2024-09-14T05:52:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2943668224 bytes 2024-09-14T05:52:15.459 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2999226368 bytes 2024-09-14T05:52:15.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 3025874944 bytes 2024-09-14T05:52:15.659 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 3035298816 bytes 2024-09-14T05:52:15.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 3042014208 bytes 2024-09-14T05:52:15.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 3084344320 bytes 2024-09-14T05:52:15.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 3108176896 bytes 2024-09-14T05:52:16.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 3168037888 bytes 2024-09-14T05:52:16.273 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 3213941760 bytes 2024-09-14T05:52:16.422 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 3274097664 bytes 2024-09-14T05:52:16.534 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 3310637056 bytes 2024-09-14T05:52:16.658 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7300 IOs, 3345089536 bytes 2024-09-14T05:52:16.770 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7400 IOs, 3359822848 bytes 2024-09-14T05:52:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7500 IOs, 3457636352 bytes 2024-09-14T05:52:17.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7600 IOs, 3500273664 bytes 2024-09-14T05:52:17.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7700 IOs, 3515771904 bytes 2024-09-14T05:52:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7800 IOs, 3536671744 bytes 2024-09-14T05:52:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7900 IOs, 3547735040 bytes 2024-09-14T05:52:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8000 IOs, 3566083072 bytes 2024-09-14T05:52:17.646 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8100 IOs, 3642757120 bytes 2024-09-14T05:52:17.779 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8200 IOs, 3691950080 bytes 2024-09-14T05:52:17.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8300 IOs, 3793376256 bytes 2024-09-14T05:52:18.084 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8400 IOs, 3825026048 bytes 2024-09-14T05:52:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8500 IOs, 3825631232 bytes 2024-09-14T05:52:18.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8600 IOs, 3886868480 bytes 2024-09-14T05:52:18.460 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8700 IOs, 3941683200 bytes 2024-09-14T05:52:18.483 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-09-14T05:52:18.483 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T05:52:18.483 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-09-14T05:52:18.484 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-14T05:52:18.484 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-14T05:52:18.487 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-14T05:52:18.487 INFO:tasks.rbd_mirror_thrash:waiting for 83 secs before thrashing 2024-09-14T05:52:18.584 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8800 IOs, 3996376064 bytes 2024-09-14T05:52:18.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8900 IOs, 4019130368 bytes 2024-09-14T05:52:18.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9000 IOs, 4054156288 bytes 2024-09-14T05:52:18.966 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9100 IOs, 4074570752 bytes 2024-09-14T05:52:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9200 IOs, 4168022016 bytes 2024-09-14T05:52:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9300 IOs, 4278402048 bytes 2024-09-14T05:52:19.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9400 IOs, 4360260608 bytes 2024-09-14T05:52:19.619 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9500 IOs, 4385083392 bytes 2024-09-14T05:52:19.787 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9600 IOs, 4468994048 bytes 2024-09-14T05:52:19.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9700 IOs, 4530600960 bytes 2024-09-14T05:52:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9800 IOs, 4553291776 bytes 2024-09-14T05:52:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9900 IOs, 4594621440 bytes 2024-09-14T05:52:20.301 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10000 IOs, 4640983040 bytes 2024-09-14T05:52:20.457 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10100 IOs, 4714150912 bytes 2024-09-14T05:52:20.558 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10200 IOs, 4724026368 bytes 2024-09-14T05:52:20.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10300 IOs, 4724652032 bytes 2024-09-14T05:52:20.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10400 IOs, 4762520576 bytes 2024-09-14T05:52:20.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:52:20.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:52:20.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:52:20.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:52:20.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:52:20.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_47 2024-09-14T05:52:20.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_47 snapshot 128M 2024-09-14T05:52:20.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:52:20.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:52:20.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:52:20.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:52:20.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_47 2024-09-14T05:52:20.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:52:20.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:52:20.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-14T05:52:20.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:52:20.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_47 128M 2024-09-14T05:52:20.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:52:20.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:52:20.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:52:20.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:52:20.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_47 2024-09-14T05:52:20.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:52:20.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T05:52:20.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-14T05:52:20.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-14T05:52:20.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:52:20.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_47 2024-09-14T05:52:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T05:52:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:52:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-14T05:52:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:52:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_47 snapshot 2024-09-14T05:52:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:52:21.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:52:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_47 2024-09-14T05:52:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:52:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_47 snapshot 2024-09-14T05:52:21.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T05:52:21.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_47 2024-09-14T05:52:21.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_47': 2024-09-14T05:52:21.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:52:21.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:52:21.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T05:52:21.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 14acb2433dd5 2024-09-14T05:52:21.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.14acb2433dd5 2024-09-14T05:52:21.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:52:21.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T05:52:21.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:52:21.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:52:21.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:52:21 2024 2024-09-14T05:52:21.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:52:21 2024 2024-09-14T05:52:21.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:52:21 2024 2024-09-14T05:52:21.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:52:21.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:52:21.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 0c24795a-37d5-4c8a-b3a8-c2ecda778c6a 2024-09-14T05:52:21.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T05:52:21.960 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:52:21.960 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_47 2024-09-14T05:52:21.960 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:52:21.960 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:52:21.960 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_47 2024-09-14T05:52:21.960 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:52:21.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=13 2024-09-14T05:52:21.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:52:21.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_47 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:52:22.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 22394880 bytes 2024-09-14T05:52:22.272 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 70542336 bytes 2024-09-14T05:52:22.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 120177664 bytes 2024-09-14T05:52:22.516 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 137080832 bytes 2024-09-14T05:52:22.599 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 138504192 bytes 2024-09-14T05:52:22.705 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 151755776 bytes 2024-09-14T05:52:22.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 175310848 bytes 2024-09-14T05:52:22.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 225740800 bytes 2024-09-14T05:52:23.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 262067200 bytes 2024-09-14T05:52:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 314201088 bytes 2024-09-14T05:52:23.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 373242880 bytes 2024-09-14T05:52:23.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 429346816 bytes 2024-09-14T05:52:23.600 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 474212352 bytes 2024-09-14T05:52:23.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 497348608 bytes 2024-09-14T05:52:23.811 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 505713664 bytes 2024-09-14T05:52:23.911 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 519682048 bytes 2024-09-14T05:52:24.055 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 564777984 bytes 2024-09-14T05:52:24.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 573789184 bytes 2024-09-14T05:52:24.250 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 591766528 bytes 2024-09-14T05:52:24.368 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 637441024 bytes 2024-09-14T05:52:24.477 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 652742656 bytes 2024-09-14T05:52:24.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 669748224 bytes 2024-09-14T05:52:24.697 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 700267520 bytes 2024-09-14T05:52:24.816 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 735550464 bytes 2024-09-14T05:52:24.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 768069632 bytes 2024-09-14T05:52:25.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 806941696 bytes 2024-09-14T05:52:25.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 830709760 bytes 2024-09-14T05:52:25.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 852298752 bytes 2024-09-14T05:52:25.460 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 945970176 bytes 2024-09-14T05:52:25.571 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 977344512 bytes 2024-09-14T05:52:25.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 998556672 bytes 2024-09-14T05:52:25.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1053040640 bytes 2024-09-14T05:52:25.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1097492480 bytes 2024-09-14T05:52:26.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1115260928 bytes 2024-09-14T05:52:26.133 INFO:tasks.ceph.cluster2.osd.1.smithi191.stderr:2024-09-14T05:52:26.132+0000 7f9375a93640 -1 reset not still connected to 0x5611e6ab0f70 2024-09-14T05:52:26.135 INFO:tasks.ceph.cluster2.osd.1.smithi191.stderr:2024-09-14T05:52:26.132+0000 7f9375a93640 -1 reset not still connected to 0x5611e6ab1930 2024-09-14T05:52:26.141 INFO:tasks.ceph.cluster2.osd.1.smithi191.stderr:2024-09-14T05:52:26.138+0000 7f9375a93640 -1 reset not still connected to 0x5611e6ab1ba0 2024-09-14T05:52:26.141 INFO:tasks.ceph.cluster2.osd.1.smithi191.stderr:2024-09-14T05:52:26.138+0000 7f9375a93640 -1 reset not still connected to 0x5611e6ae6ea0 2024-09-14T05:52:26.141 INFO:tasks.ceph.cluster2.osd.1.smithi191.stderr:2024-09-14T05:52:26.138+0000 7f9375a93640 -1 reset not still connected to 0x5611e6ae71e0 2024-09-14T05:52:26.141 INFO:tasks.ceph.cluster2.osd.1.smithi191.stderr:2024-09-14T05:52:26.138+0000 7f9375a93640 -1 reset not still connected to 0x5611e752b860 2024-09-14T05:52:26.141 INFO:tasks.ceph.cluster2.osd.1.smithi191.stderr:2024-09-14T05:52:26.138+0000 7f9375a93640 -1 reset not still connected to 0x5611e94245b0 2024-09-14T05:52:26.142 INFO:tasks.ceph.cluster2.osd.1.smithi191.stderr:2024-09-14T05:52:26.138+0000 7f9375a93640 -1 reset not still connected to 0x5611eb96a1a0 2024-09-14T05:52:26.142 INFO:tasks.ceph.cluster2.osd.1.smithi191.stderr:2024-09-14T05:52:26.138+0000 7f9375a93640 -1 reset not still connected to 0x5611ec465e10 2024-09-14T05:52:26.142 INFO:tasks.ceph.cluster2.osd.1.smithi191.stderr:2024-09-14T05:52:26.138+0000 7f9375a93640 -1 reset not still connected to 0x5611f3a34410 2024-09-14T05:52:26.142 INFO:tasks.ceph.cluster2.osd.1.smithi191.stderr:2024-09-14T05:52:26.138+0000 7f9375a93640 -1 reset not still connected to 0x5612056b72b0 2024-09-14T05:52:26.149 INFO:tasks.ceph.cluster2.osd.1.smithi191.stderr:2024-09-14T05:52:26.145+0000 7f9375a93640 -1 reset not still connected to 0x56120fde2f70 2024-09-14T05:52:26.149 INFO:tasks.ceph.cluster2.osd.1.smithi191.stderr:2024-09-14T05:52:26.145+0000 7f9375a93640 -1 reset not still connected to 0x561212fa48f0 2024-09-14T05:52:26.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1142672384 bytes 2024-09-14T05:52:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1147432960 bytes 2024-09-14T05:52:26.426 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1222410240 bytes 2024-09-14T05:52:26.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1238674432 bytes 2024-09-14T05:52:26.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1267094528 bytes 2024-09-14T05:52:26.728 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1267728384 bytes 2024-09-14T05:52:26.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1320471552 bytes 2024-09-14T05:52:26.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1355333632 bytes 2024-09-14T05:52:27.161 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1455695872 bytes 2024-09-14T05:52:27.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1481708544 bytes 2024-09-14T05:52:27.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1537199104 bytes 2024-09-14T05:52:27.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1541569536 bytes 2024-09-14T05:52:27.654 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 1601677312 bytes 2024-09-14T05:52:27.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 1665635328 bytes 2024-09-14T05:52:27.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 1701544960 bytes 2024-09-14T05:52:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 1816367104 bytes 2024-09-14T05:52:28.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 1838695424 bytes 2024-09-14T05:52:28.325 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 1852806144 bytes 2024-09-14T05:52:28.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 1928833024 bytes 2024-09-14T05:52:28.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 1981449216 bytes 2024-09-14T05:52:28.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2032489472 bytes 2024-09-14T05:52:29.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2065126400 bytes 2024-09-14T05:52:29.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2083521536 bytes 2024-09-14T05:52:29.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2106600448 bytes 2024-09-14T05:52:29.478 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2230174720 bytes 2024-09-14T05:52:29.615 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2267660288 bytes 2024-09-14T05:52:29.744 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2313912320 bytes 2024-09-14T05:52:29.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2358804480 bytes 2024-09-14T05:52:30.038 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2439054336 bytes 2024-09-14T05:52:30.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2509391872 bytes 2024-09-14T05:52:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2521725952 bytes 2024-09-14T05:52:30.400 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 2543954944 bytes 2024-09-14T05:52:30.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 2589210624 bytes 2024-09-14T05:52:30.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 2632165376 bytes 2024-09-14T05:52:30.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 2696770560 bytes 2024-09-14T05:52:30.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 2719934464 bytes 2024-09-14T05:52:31.068 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 2790426624 bytes 2024-09-14T05:52:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 2829268992 bytes 2024-09-14T05:52:31.325 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7300 IOs, 2885897216 bytes 2024-09-14T05:52:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7400 IOs, 2923513856 bytes 2024-09-14T05:52:31.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7500 IOs, 2973211648 bytes 2024-09-14T05:52:31.736 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7600 IOs, 3049271296 bytes 2024-09-14T05:52:31.859 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7700 IOs, 3088252928 bytes 2024-09-14T05:52:31.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7800 IOs, 3108690944 bytes 2024-09-14T05:52:32.067 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7900 IOs, 3129040896 bytes 2024-09-14T05:52:32.190 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8000 IOs, 3169051648 bytes 2024-09-14T05:52:32.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8100 IOs, 3173158912 bytes 2024-09-14T05:52:32.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8200 IOs, 3208845312 bytes 2024-09-14T05:52:32.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8300 IOs, 3234661376 bytes 2024-09-14T05:52:32.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8400 IOs, 3235256320 bytes 2024-09-14T05:52:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8500 IOs, 3260805120 bytes 2024-09-14T05:52:32.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8600 IOs, 3342694400 bytes 2024-09-14T05:52:32.998 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8700 IOs, 3392604160 bytes 2024-09-14T05:52:33.085 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8800 IOs, 3393239040 bytes 2024-09-14T05:52:33.187 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8900 IOs, 3413104640 bytes 2024-09-14T05:52:33.299 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9000 IOs, 3440334848 bytes 2024-09-14T05:52:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9100 IOs, 3462278144 bytes 2024-09-14T05:52:33.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9200 IOs, 3475908608 bytes 2024-09-14T05:52:33.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9300 IOs, 3536579584 bytes 2024-09-14T05:52:33.669 INFO:tasks.ceph.cluster2.osd.2.smithi191.stderr:2024-09-14T05:52:33.666+0000 7f3763311640 -1 reset not still connected to 0x55facb083a00 2024-09-14T05:52:33.673 INFO:tasks.ceph.cluster2.osd.2.smithi191.stderr:2024-09-14T05:52:33.669+0000 7f3763311640 -1 reset not still connected to 0x55facb3d36c0 2024-09-14T05:52:33.673 INFO:tasks.ceph.cluster2.osd.2.smithi191.stderr:2024-09-14T05:52:33.669+0000 7f3763311640 -1 reset not still connected to 0x55facbcfc1a0 2024-09-14T05:52:33.673 INFO:tasks.ceph.cluster2.osd.2.smithi191.stderr:2024-09-14T05:52:33.670+0000 7f3763311640 -1 reset not still connected to 0x55facbd83860 2024-09-14T05:52:33.673 INFO:tasks.ceph.cluster2.osd.2.smithi191.stderr:2024-09-14T05:52:33.670+0000 7f3763311640 -1 reset not still connected to 0x55facc577a00 2024-09-14T05:52:33.673 INFO:tasks.ceph.cluster2.osd.2.smithi191.stderr:2024-09-14T05:52:33.670+0000 7f3763311640 -1 reset not still connected to 0x55facdb99040 2024-09-14T05:52:33.673 INFO:tasks.ceph.cluster2.osd.2.smithi191.stderr:2024-09-14T05:52:33.670+0000 7f3763311640 -1 reset not still connected to 0x55facb083a00 2024-09-14T05:52:33.673 INFO:tasks.ceph.cluster2.osd.2.smithi191.stderr:2024-09-14T05:52:33.670+0000 7f3763311640 -1 reset not still connected to 0x55facdb99040 2024-09-14T05:52:33.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9400 IOs, 3555666944 bytes 2024-09-14T05:52:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9500 IOs, 3576782848 bytes 2024-09-14T05:52:33.969 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9600 IOs, 3599512576 bytes 2024-09-14T05:52:34.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9700 IOs, 3617203200 bytes 2024-09-14T05:52:34.193 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9800 IOs, 3657333760 bytes 2024-09-14T05:52:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9900 IOs, 3712074752 bytes 2024-09-14T05:52:34.499 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10000 IOs, 3807685632 bytes 2024-09-14T05:52:34.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10100 IOs, 3829494784 bytes 2024-09-14T05:52:34.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10200 IOs, 3855602688 bytes 2024-09-14T05:52:34.816 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10300 IOs, 3873158144 bytes 2024-09-14T05:52:34.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:52:34.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:52:34.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:52:34.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:52:34.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:52:34.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_48 2024-09-14T05:52:34.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_48 snapshot 128M 2024-09-14T05:52:34.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:52:34.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:52:34.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:52:34.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:52:34.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_48 2024-09-14T05:52:34.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:52:34.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:52:34.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-14T05:52:34.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:52:34.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_48 128M 2024-09-14T05:52:34.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:52:34.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:52:34.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:52:34.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:52:34.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_48 2024-09-14T05:52:34.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:52:34.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T05:52:34.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-14T05:52:34.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-14T05:52:34.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:52:34.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_48 2024-09-14T05:52:35.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T05:52:35.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:52:35.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-14T05:52:35.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:52:35.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_48 snapshot 2024-09-14T05:52:35.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:52:35.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:52:35.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_48 2024-09-14T05:52:35.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:52:35.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_48 snapshot 2024-09-14T05:52:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T05:52:35.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_48 2024-09-14T05:52:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_48': 2024-09-14T05:52:35.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:52:35.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:52:35.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T05:52:35.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 14b8b2200d92 2024-09-14T05:52:35.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.14b8b2200d92 2024-09-14T05:52:35.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:52:35.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T05:52:35.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:52:35.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:52:35.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:52:35 2024 2024-09-14T05:52:35.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:52:35 2024 2024-09-14T05:52:35.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:52:35 2024 2024-09-14T05:52:35.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:52:35.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:52:35.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 630369a5-7829-494e-9cd3-0d8a963a0ae4 2024-09-14T05:52:35.982 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T05:52:35.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:52:35.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_48 2024-09-14T05:52:35.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:52:35.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:52:35.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_48 2024-09-14T05:52:35.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:52:35.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=14 2024-09-14T05:52:35.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:52:35.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_48 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:52:36.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 73337856 bytes 2024-09-14T05:52:36.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 91078656 bytes 2024-09-14T05:52:36.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 142991360 bytes 2024-09-14T05:52:36.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 166169600 bytes 2024-09-14T05:52:36.699 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 205768704 bytes 2024-09-14T05:52:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 219048960 bytes 2024-09-14T05:52:36.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 239000576 bytes 2024-09-14T05:52:37.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 262519808 bytes 2024-09-14T05:52:37.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 306620416 bytes 2024-09-14T05:52:37.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 380540928 bytes 2024-09-14T05:52:37.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 422573056 bytes 2024-09-14T05:52:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 455198720 bytes 2024-09-14T05:52:37.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 518836224 bytes 2024-09-14T05:52:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 584608768 bytes 2024-09-14T05:52:38.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 637018112 bytes 2024-09-14T05:52:38.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 714373120 bytes 2024-09-14T05:52:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 746124288 bytes 2024-09-14T05:52:38.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 787206144 bytes 2024-09-14T05:52:38.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 870106112 bytes 2024-09-14T05:52:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 917551104 bytes 2024-09-14T05:52:38.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 989139968 bytes 2024-09-14T05:52:39.019 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 1040368640 bytes 2024-09-14T05:52:39.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1042014208 bytes 2024-09-14T05:52:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1101612032 bytes 2024-09-14T05:52:39.384 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1147409408 bytes 2024-09-14T05:52:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1178632192 bytes 2024-09-14T05:52:39.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1250086912 bytes 2024-09-14T05:52:39.795 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1302529024 bytes 2024-09-14T05:52:39.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1350121472 bytes 2024-09-14T05:52:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1397344256 bytes 2024-09-14T05:52:40.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1460027392 bytes 2024-09-14T05:52:40.422 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1577826304 bytes 2024-09-14T05:52:40.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1599028224 bytes 2024-09-14T05:52:40.699 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1654996992 bytes 2024-09-14T05:52:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1668025344 bytes 2024-09-14T05:52:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1708253184 bytes 2024-09-14T05:52:41.073 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1795445760 bytes 2024-09-14T05:52:41.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1814277120 bytes 2024-09-14T05:52:41.330 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1855928320 bytes 2024-09-14T05:52:41.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1916238848 bytes 2024-09-14T05:52:41.631 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1982264320 bytes 2024-09-14T05:52:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 2009363456 bytes 2024-09-14T05:52:41.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 2077908992 bytes 2024-09-14T05:52:42.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 2098187264 bytes 2024-09-14T05:52:42.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 2132584448 bytes 2024-09-14T05:52:42.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 2167263232 bytes 2024-09-14T05:52:42.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 2168145920 bytes 2024-09-14T05:52:42.523 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2258532352 bytes 2024-09-14T05:52:42.643 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2302797824 bytes 2024-09-14T05:52:42.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2379552768 bytes 2024-09-14T05:52:42.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2392051712 bytes 2024-09-14T05:52:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2423291904 bytes 2024-09-14T05:52:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2472850432 bytes 2024-09-14T05:52:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2489423872 bytes 2024-09-14T05:52:43.458 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2568057856 bytes 2024-09-14T05:52:43.625 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2643278848 bytes 2024-09-14T05:52:43.747 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2657967104 bytes 2024-09-14T05:52:43.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2696198144 bytes 2024-09-14T05:52:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2777758720 bytes 2024-09-14T05:52:44.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2830217216 bytes 2024-09-14T05:52:44.295 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2851793920 bytes 2024-09-14T05:52:44.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2891010048 bytes 2024-09-14T05:52:44.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2926114816 bytes 2024-09-14T05:52:44.652 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2950031360 bytes 2024-09-14T05:52:44.802 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 3012883456 bytes 2024-09-14T05:52:44.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 3013513216 bytes 2024-09-14T05:52:45.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 3046067200 bytes 2024-09-14T05:52:45.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 3077890048 bytes 2024-09-14T05:52:45.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 3106617344 bytes 2024-09-14T05:52:45.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 3175965696 bytes 2024-09-14T05:52:45.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 3197256704 bytes 2024-09-14T05:52:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 3207142400 bytes 2024-09-14T05:52:45.738 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7300 IOs, 3255964672 bytes 2024-09-14T05:52:45.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7400 IOs, 3339479040 bytes 2024-09-14T05:52:45.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7500 IOs, 3346205696 bytes 2024-09-14T05:52:46.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7600 IOs, 3413492736 bytes 2024-09-14T05:52:46.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7700 IOs, 3465636864 bytes 2024-09-14T05:52:46.406 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7800 IOs, 3506989056 bytes 2024-09-14T05:52:46.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7900 IOs, 3550685184 bytes 2024-09-14T05:52:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8000 IOs, 3583288320 bytes 2024-09-14T05:52:46.798 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8100 IOs, 3643879424 bytes 2024-09-14T05:52:46.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8200 IOs, 3645361152 bytes 2024-09-14T05:52:47.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8300 IOs, 3723456512 bytes 2024-09-14T05:52:47.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8400 IOs, 3761629184 bytes 2024-09-14T05:52:47.270 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8500 IOs, 3802566656 bytes 2024-09-14T05:52:47.426 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8600 IOs, 3845514240 bytes 2024-09-14T05:52:47.524 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8700 IOs, 3876567040 bytes 2024-09-14T05:52:47.692 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8800 IOs, 3943072768 bytes 2024-09-14T05:52:47.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8900 IOs, 3981980672 bytes 2024-09-14T05:52:47.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9000 IOs, 4006247424 bytes 2024-09-14T05:52:48.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9100 IOs, 4064125952 bytes 2024-09-14T05:52:48.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9200 IOs, 4073720832 bytes 2024-09-14T05:52:48.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9300 IOs, 4097009664 bytes 2024-09-14T05:52:48.407 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9400 IOs, 4140510208 bytes 2024-09-14T05:52:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9500 IOs, 4172268544 bytes 2024-09-14T05:52:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9600 IOs, 4218881024 bytes 2024-09-14T05:52:48.798 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9700 IOs, 4273349632 bytes 2024-09-14T05:52:48.892 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9800 IOs, 4278230016 bytes 2024-09-14T05:52:49.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9900 IOs, 4322206720 bytes 2024-09-14T05:52:49.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10000 IOs, 4367923200 bytes 2024-09-14T05:52:49.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10100 IOs, 4405715968 bytes 2024-09-14T05:52:49.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10200 IOs, 4454375424 bytes 2024-09-14T05:52:49.545 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10300 IOs, 4501501952 bytes 2024-09-14T05:52:49.656 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10400 IOs, 4527092736 bytes 2024-09-14T05:52:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10500 IOs, 4589014016 bytes 2024-09-14T05:52:49.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10600 IOs, 4634732544 bytes 2024-09-14T05:52:49.991 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:52:49.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:52:49.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:52:49.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:52:49.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:52:49.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_49 2024-09-14T05:52:49.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_49 snapshot 128M 2024-09-14T05:52:49.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:52:49.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:52:49.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:52:49.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:52:49.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_49 2024-09-14T05:52:49.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:52:49.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:52:49.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-14T05:52:49.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:52:49.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_49 128M 2024-09-14T05:52:49.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:52:49.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:52:49.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:52:49.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:52:49.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_49 2024-09-14T05:52:49.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:52:49.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T05:52:49.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-14T05:52:49.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-14T05:52:49.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:52:49.995 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_49 2024-09-14T05:52:50.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T05:52:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:52:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-14T05:52:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:52:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_49 snapshot 2024-09-14T05:52:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:52:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:52:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_49 2024-09-14T05:52:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:52:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_49 snapshot 2024-09-14T05:52:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T05:52:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_49 2024-09-14T05:52:51.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_49': 2024-09-14T05:52:51.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:52:51.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:52:51.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T05:52:51.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 14c4f5adf1a8 2024-09-14T05:52:51.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.14c4f5adf1a8 2024-09-14T05:52:51.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:52:51.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T05:52:51.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:52:51.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:52:51.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:52:50 2024 2024-09-14T05:52:51.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:52:50 2024 2024-09-14T05:52:51.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:52:50 2024 2024-09-14T05:52:51.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:52:51.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:52:51.011 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 2567c386-672b-4b12-ae9d-ec5af9fc0c43 2024-09-14T05:52:51.011 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T05:52:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:52:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_49 2024-09-14T05:52:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:52:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:52:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_49 2024-09-14T05:52:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:52:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=6 2024-09-14T05:52:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:52:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_49 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:52:51.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 27480064 bytes 2024-09-14T05:52:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 65263616 bytes 2024-09-14T05:52:51.470 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 103661568 bytes 2024-09-14T05:52:51.625 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 179604480 bytes 2024-09-14T05:52:51.795 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 264818688 bytes 2024-09-14T05:52:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 358376448 bytes 2024-09-14T05:52:52.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 421850112 bytes 2024-09-14T05:52:52.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 458425344 bytes 2024-09-14T05:52:52.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 521380864 bytes 2024-09-14T05:52:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 551779328 bytes 2024-09-14T05:52:52.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 582991872 bytes 2024-09-14T05:52:52.771 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 645224448 bytes 2024-09-14T05:52:52.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 660321280 bytes 2024-09-14T05:52:52.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 678042624 bytes 2024-09-14T05:52:53.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 741639168 bytes 2024-09-14T05:52:53.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 767657984 bytes 2024-09-14T05:52:53.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 825611264 bytes 2024-09-14T05:52:53.524 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 888393728 bytes 2024-09-14T05:52:53.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 940332032 bytes 2024-09-14T05:52:53.779 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 962461696 bytes 2024-09-14T05:52:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 1006008320 bytes 2024-09-14T05:52:54.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 1069287424 bytes 2024-09-14T05:52:54.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1069894656 bytes 2024-09-14T05:52:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1103015936 bytes 2024-09-14T05:52:54.382 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1157346304 bytes 2024-09-14T05:52:54.515 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1173857280 bytes 2024-09-14T05:52:54.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1246543872 bytes 2024-09-14T05:52:54.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1297310720 bytes 2024-09-14T05:52:54.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1339236352 bytes 2024-09-14T05:52:55.054 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1365935104 bytes 2024-09-14T05:52:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1407534080 bytes 2024-09-14T05:52:55.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1413776384 bytes 2024-09-14T05:52:55.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1426376704 bytes 2024-09-14T05:52:55.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1474878464 bytes 2024-09-14T05:52:55.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1547235328 bytes 2024-09-14T05:52:55.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1565950976 bytes 2024-09-14T05:52:55.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1624798208 bytes 2024-09-14T05:52:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1658037248 bytes 2024-09-14T05:52:56.095 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1658651648 bytes 2024-09-14T05:52:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1685664768 bytes 2024-09-14T05:52:56.325 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1704684544 bytes 2024-09-14T05:52:56.436 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1730204672 bytes 2024-09-14T05:52:56.559 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1775156224 bytes 2024-09-14T05:52:56.712 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1858743296 bytes 2024-09-14T05:52:56.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1926496256 bytes 2024-09-14T05:52:56.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1945116672 bytes 2024-09-14T05:52:57.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:52:57.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:52:57.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:52:57.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:52:57.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:52:57.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_50 2024-09-14T05:52:57.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image_and_enable_mirror cluster2 mirror image_50 snapshot 128M 2024-09-14T05:52:57.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:52:57.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:52:57.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:52:57.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:52:57.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_50 2024-09-14T05:52:57.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:52:57.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:52:57.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n snapshot ']' 2024-09-14T05:52:57.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:52:57.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_50 128M 2024-09-14T05:52:57.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:52:57.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:52:57.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:52:57.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:52:57.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_50 2024-09-14T05:52:57.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:52:57.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T05:52:57.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 128M ']' 2024-09-14T05:52:57.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=128M 2024-09-14T05:52:57.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T05:52:57.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_50 2024-09-14T05:52:57.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ is_pool_mirror_mode_image mirror 2024-09-14T05:52:57.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:52:57.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' image = image ']' 2024-09-14T05:52:57.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:52:57.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_50 snapshot 2024-09-14T05:52:57.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:52:57.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:52:57.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_50 2024-09-14T05:52:57.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T05:52:57.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_50 snapshot 2024-09-14T05:52:57.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T05:52:57.991 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_50 2024-09-14T05:52:58.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_50': 2024-09-14T05:52:58.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:52:58.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:52:58.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T05:52:58.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 14d0a8c109d4 2024-09-14T05:52:58.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.14d0a8c109d4 2024-09-14T05:52:58.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:52:58.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T05:52:58.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:52:58.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:52:58.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:52:57 2024 2024-09-14T05:52:58.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:52:57 2024 2024-09-14T05:52:58.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:52:57 2024 2024-09-14T05:52:58.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:52:58.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:52:58.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: c89c328d-a242-48de-b9a9-130c74b3f877 2024-09-14T05:52:58.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T05:52:58.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n '' ']' 2024-09-14T05:52:58.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_50 2024-09-14T05:52:58.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:52:58.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:52:58.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_50 2024-09-14T05:52:58.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T05:52:58.035 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=5 2024-09-14T05:52:58.035 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T05:52:58.036 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_50 --debug-rbd=20 --debug-journaler=20 2024-09-14T05:52:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 15125504 bytes 2024-09-14T05:52:58.333 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 48656384 bytes 2024-09-14T05:52:58.465 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 107173888 bytes 2024-09-14T05:52:58.498 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.4758581040190636) > weight (0.25) 2024-09-14T05:52:58.498 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5174453819948793) > weight (0.25) 2024-09-14T05:52:58.498 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-09-14T05:52:58.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-09-14T05:52:58.499 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2024-09-14T05:52:58.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-14T05:52:58.498+0000 7fd78a0e6640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 44412) UID: 1000 2024-09-14T05:52:58.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 155847680 bytes 2024-09-14T05:52:58.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 203650048 bytes 2024-09-14T05:52:58.892 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 249656320 bytes 2024-09-14T05:52:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 301119488 bytes 2024-09-14T05:52:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 335464448 bytes 2024-09-14T05:52:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 353775616 bytes 2024-09-14T05:52:59.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 466114560 bytes 2024-09-14T05:52:59.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 522902528 bytes 2024-09-14T05:52:59.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 583814144 bytes 2024-09-14T05:52:59.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 622726144 bytes 2024-09-14T05:52:59.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 647404544 bytes 2024-09-14T05:53:00.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 668723200 bytes 2024-09-14T05:53:00.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 675097600 bytes 2024-09-14T05:53:00.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 689804288 bytes 2024-09-14T05:53:00.398 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 749582336 bytes 2024-09-14T05:53:00.500 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-09-14T05:53:00.500 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T05:53:00.500 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-09-14T05:53:00.501 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-14T05:53:00.501 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-14T05:53:00.504 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-14T05:53:00.504 INFO:tasks.rbd_mirror_thrash:waiting for 91 secs before thrashing 2024-09-14T05:53:00.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 793894912 bytes 2024-09-14T05:53:00.657 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 823783424 bytes 2024-09-14T05:53:00.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 893343744 bytes 2024-09-14T05:53:00.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 906515456 bytes 2024-09-14T05:53:01.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 932118528 bytes 2024-09-14T05:53:01.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 961145856 bytes 2024-09-14T05:53:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 988952576 bytes 2024-09-14T05:53:01.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1061169152 bytes 2024-09-14T05:53:01.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1073222656 bytes 2024-09-14T05:53:01.624 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1091993600 bytes 2024-09-14T05:53:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1203753984 bytes 2024-09-14T05:53:01.929 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1233538048 bytes 2024-09-14T05:53:02.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1235954688 bytes 2024-09-14T05:53:02.117 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1248193536 bytes 2024-09-14T05:53:02.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1278988288 bytes 2024-09-14T05:53:02.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1327590400 bytes 2024-09-14T05:53:02.506 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1391168512 bytes 2024-09-14T05:53:02.631 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1436284928 bytes 2024-09-14T05:53:02.730 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1440508928 bytes 2024-09-14T05:53:02.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T05:53:02.778+0000 7f1bcbab3640 -1 rbd::mirror::ImageReplayer: 0x55710d360f00 [3/3f054947-dd92-44d7-bb13-91eab274073e] start: already running 2024-09-14T05:53:02.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T05:53:02.778+0000 7f1bcbab3640 -1 rbd::mirror::ImageReplayer: 0x55710d308280 [3/5962be87-a0c9-4726-93e1-f3aad0bd9173] start: already running 2024-09-14T05:53:02.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T05:53:02.778+0000 7f1bcbab3640 -1 rbd::mirror::ImageReplayer: 0x55710d309680 [3/71b41da4-b167-4eb2-a0ea-67c283cfec10] start: already running 2024-09-14T05:53:02.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T05:53:02.778+0000 7f1bcbab3640 -1 rbd::mirror::ImageReplayer: 0x55710d33e280 [3/847da547-970e-41a0-bee0-110873986de0] start: already running 2024-09-14T05:53:02.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T05:53:02.778+0000 7f1bcbab3640 -1 rbd::mirror::ImageReplayer: 0x55710d308500 [3/5c00a5fc-12ce-46ba-b4c0-0e6aa090fe1a] start: already running 2024-09-14T05:53:02.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T05:53:02.778+0000 7f1bcbab3640 -1 rbd::mirror::ImageReplayer: 0x55710d360280 [3/5049b9ce-c032-4812-91d0-7dbca1b8b176] start: already running 2024-09-14T05:53:02.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T05:53:02.778+0000 7f1bcbab3640 -1 rbd::mirror::ImageReplayer: 0x55710d309180 [3/6654bddc-72e9-47b0-8061-615cff98334f] start: already running 2024-09-14T05:53:02.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T05:53:02.778+0000 7f1bcbab3640 -1 rbd::mirror::ImageReplayer: 0x55710d309b80 [3/7b610a90-6bb9-4ffb-8270-bbf747673d2a] start: already running 2024-09-14T05:53:02.780 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T05:53:02.778+0000 7f1bcbab3640 -1 rbd::mirror::ImageReplayer: 0x55710d308a00 [3/6249a2fd-1afb-4dee-89ca-39ba30cbe74d] start: already running 2024-09-14T05:53:02.781 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T05:53:02.778+0000 7f1bcbab3640 -1 rbd::mirror::ImageReplayer: 0x55710d308000 [3/53467864-fd16-454f-8435-46f986b18a2f] start: already running 2024-09-14T05:53:02.781 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T05:53:02.778+0000 7f1bcbab3640 -1 rbd::mirror::ImageReplayer: 0x55710d308f00 [3/662fa768-1dea-4f15-91bf-edf297ebd84a] start: already running 2024-09-14T05:53:02.781 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T05:53:02.778+0000 7f1bcbab3640 -1 rbd::mirror::ImageReplayer: 0x55710d361680 [3/4cab209c-76f4-435e-9f27-921251cc9286] start: already running 2024-09-14T05:53:02.781 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T05:53:02.778+0000 7f1bcbab3640 -1 rbd::mirror::ImageReplayer: 0x55710d308780 [3/5c71f4af-d4cf-4db0-96fd-a68b7ed67897] start: already running 2024-09-14T05:53:02.781 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T05:53:02.778+0000 7f1bcbab3640 -1 rbd::mirror::ImageReplayer: 0x55710d33e000 [3/81e2e849-43ea-4a3d-825a-ccda8c67058a] start: already running 2024-09-14T05:53:02.781 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T05:53:02.778+0000 7f1bcbab3640 -1 rbd::mirror::ImageReplayer: 0x55710d309400 [3/702a468e-cfaa-4567-a91a-60fa43b31b1f] start: already running 2024-09-14T05:53:02.781 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T05:53:02.778+0000 7f1bcbab3640 -1 rbd::mirror::ImageReplayer: 0x55710d309900 [3/72651fba-3b42-454d-a354-6f5b7be1dd8d] start: already running 2024-09-14T05:53:02.796 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T05:53:02.794+0000 7f1bcb2b2640 -1 rbd::mirror::ImageReplayer: 0x55710d361900 [3/8d3c3903-631c-42c7-8d30-a5b307b883eb] start: already running 2024-09-14T05:53:02.796 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T05:53:02.794+0000 7f1bcbab3640 -1 rbd::mirror::ImageReplayer: 0x55710d360a00 [3/37b3b339-d8a4-4dd0-ac91-56b0d1df8ebd] start: already running 2024-09-14T05:53:02.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T05:53:02.810+0000 7f1bcbab3640 -1 rbd::mirror::ImageReplayer: 0x55710d308c80 [3/630369a5-7829-494e-9cd3-0d8a963a0ae4] start: already running 2024-09-14T05:53:02.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1475986432 bytes 2024-09-14T05:53:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1512681472 bytes 2024-09-14T05:53:03.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T05:53:03.076 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T05:53:03.076 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T05:53:03.076 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:53:03.076 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_pool_images cluster2 mirror 50 2024-09-14T05:53:03.076 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:53:03.076 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:03.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_count=50 2024-09-14T05:53:03.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:53:03.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local count 2024-09-14T05:53:03.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local last_count=0 2024-09-14T05:53:03.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:53:03.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ seq 1 40 2024-09-14T05:53:03.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in `seq 1 40` 2024-09-14T05:53:03.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 1 -ne 1 2024-09-14T05:53:03.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep 'images: ' 2024-09-14T05:53:03.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cut '-d ' -f 2 2024-09-14T05:53:03.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-09-14T05:53:03.085 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: failed to query services: (13) Permission denied 2024-09-14T05:53:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ count=50 2024-09-14T05:53:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 50 = 50 2024-09-14T05:53:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:53:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_pool_healthy cluster2 mirror 2024-09-14T05:53:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:53:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:53:03.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state 2024-09-14T05:53:03.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ seq 1 40 2024-09-14T05:53:03.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in `seq 1 40` 2024-09-14T05:53:03.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 1 -ne 1 2024-09-14T05:53:03.091 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-09-14T05:53:03.091 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep 'image health:' 2024-09-14T05:53:03.091 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cut '-d ' -f 3 2024-09-14T05:53:03.128 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: failed to query services: (13) Permission denied 2024-09-14T05:53:03.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ state=OK 2024-09-14T05:53:03.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test OK = ERROR 2024-09-14T05:53:03.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test OK = OK 2024-09-14T05:53:03.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:53:03.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_pool_images cluster1 mirror 50 2024-09-14T05:53:03.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:53:03.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:03.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_count=50 2024-09-14T05:53:03.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:53:03.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local count 2024-09-14T05:53:03.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local last_count=0 2024-09-14T05:53:03.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:53:03.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ seq 1 40 2024-09-14T05:53:03.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in `seq 1 40` 2024-09-14T05:53:03.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 1 -ne 1 2024-09-14T05:53:03.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-14T05:53:03.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep 'images: ' 2024-09-14T05:53:03.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cut '-d ' -f 2 2024-09-14T05:53:03.171 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: failed to query services: (13) Permission denied 2024-09-14T05:53:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ count=50 2024-09-14T05:53:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 50 = 50 2024-09-14T05:53:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:53:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_pool_healthy cluster1 mirror 2024-09-14T05:53:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:53:03.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:03.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:53:03.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state 2024-09-14T05:53:03.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ seq 1 40 2024-09-14T05:53:03.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in `seq 1 40` 2024-09-14T05:53:03.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 1 -ne 1 2024-09-14T05:53:03.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-14T05:53:03.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep 'image health:' 2024-09-14T05:53:03.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cut '-d ' -f 3 2024-09-14T05:53:03.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: failed to query services: (13) Permission denied 2024-09-14T05:53:03.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ state=OK 2024-09-14T05:53:03.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test OK = ERROR 2024-09-14T05:53:03.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test OK = OK 2024-09-14T05:53:03.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:53:03.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ testlog 'TEST: compare many images' 2024-09-14T05:53:03.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ tee -a /tmp/tmp.sXArEbIrwv/rbd-mirror.test.log 2024-09-14T05:53:03.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ date '+%F %T' 2024-09-14T05:53:03.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo 2024-09-14 05:53:03 TEST: compare many images 2024-09-14T05:53:03.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:2024-09-14 05:53:03 TEST: compare many images 2024-09-14T05:53:03.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ seq 1 50 2024-09-14T05:53:03.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:53:03.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_1 2024-09-14T05:53:03.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_1 snap 2024-09-14T05:53:03.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:53:03.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:03.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-14T05:53:03.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:53:03.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_1@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T05:53:04.996 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_1 2024-09-14T05:53:04.996 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:53:04.996 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:04.996 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-14T05:53:04.996 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_1 started 2024-09-14T05:53:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:53:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-14T05:53:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:53:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:53:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:53:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:53:05.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_1 started 2024-09-14T05:53:05.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:53:05.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:05.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-14T05:53:05.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:53:05.998 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:53:05.998 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:53:05.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_1 2024-09-14T05:53:05.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:53:05.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:53:05.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:53:05.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:53:05.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:53:05.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:53:05.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:53:06.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:53:06.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:53:06.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:53:06.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:53:06.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_1 2024-09-14T05:53:06.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:53:06.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:53:06.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:53:06.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:53:06.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:53:06.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:53:06.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:53:06.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:53:06.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:53:06.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:53:06.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:53:06.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:53:06.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:53:06.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:53:06.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:53:06.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:53:06.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:53:06.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:53:06.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:53:06.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:53:06.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:53:06.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:53:06.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:53:06.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:53:06.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:53:06.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:53:06.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:53:06.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:53:06.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:53:06.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:53:06.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:53:06.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:53:06.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:53:06.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_1 2024-09-14T05:53:06.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_1'] 2024-09-14T05:53:06.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:53:06.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:53:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:53:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_1 2024-09-14T05:53:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-14T05:53:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:53:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:53:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T05:53:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-14T05:53:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:53:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:53:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:53:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-14T05:53:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T05:53:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:53:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:53:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-14T05:53:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:53:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T05:53:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-14T05:53:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:53:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:53:06.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T05:53:06.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-14T05:53:06.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:53:06.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:53:06.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:53:06.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:53:06.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T05:53:06.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:53:06.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:53:06.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-14T05:53:06.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-14T05:53:06.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T05:53:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:53:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:53:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:53:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_1 2024-09-14T05:53:06.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_1'] 2024-09-14T05:53:06.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:53:06.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:53:06.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:53:06.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_1 2024-09-14T05:53:06.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-14T05:53:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:53:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:53:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T05:53:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-14T05:53:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:53:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:53:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:53:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-14T05:53:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T05:53:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:53:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:53:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-14T05:53:06.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:53:06.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T05:53:06.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-14T05:53:06.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:53:06.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:53:06.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T05:53:06.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-14T05:53:06.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:53:06.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:53:06.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:53:06.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:53:06.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T05:53:06.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:53:06.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:53:06.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-14T05:53:06.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-14T05:53:06.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T05:53:06.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:53:06.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:53:06.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:53:06.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_1 2024-09-14T05:53:06.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:53:06.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:53:06.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:53:06.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:53:06.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:53:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:53:06.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:53:06.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:53:06.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_1 2024-09-14T05:53:06.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:53:06.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:53:06.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:06.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-14T05:53:06.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:53:06.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:53:06.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_1 2024-09-14T05:53:06.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:53:06.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:53:06.343 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:06.343 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-14T05:53:06.343 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_1.status 2024-09-14T05:53:06.343 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_1.status 2024-09-14T05:53:06.343 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:53:06.343 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_1.status 2024-09-14T05:53:06.343 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_1.status 2024-09-14T05:53:06.343 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_1.status 2024-09-14T05:53:06.343 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:53:06.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_1.status 2024-09-14T05:53:06.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_1 2024-09-14T05:53:06.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:53:06.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:06.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-14T05:53:06.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_1 2024-09-14T05:53:07.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 86 2024-09-14T05:53:07.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_1 /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_1.status 2024-09-14T05:53:07.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:53:07.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:07.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-14T05:53:07.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_1.status 2024-09-14T05:53:07.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_1 --format xml 2024-09-14T05:53:07.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:53:07.062 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:53:07.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=86 2024-09-14T05:53:07.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:53:07.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:53:07.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:53:07.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_1.status 2024-09-14T05:53:07.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:53:07.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:07.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-14T05:53:07.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_1.status 2024-09-14T05:53:07.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-09-14T05:53:07.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:53:07.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:53:07.323 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=35 2024-09-14T05:53:07.323 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 86 = 35 2024-09-14T05:53:07.323 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:53:07.324 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:53:07.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_1.status 2024-09-14T05:53:07.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:53:07.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:07.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-14T05:53:07.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_1.status 2024-09-14T05:53:07.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-09-14T05:53:07.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:53:07.779 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:53:07.779 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:53:07.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:53:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:53:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:53:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:53:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 86 = '' 2024-09-14T05:53:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:53:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:53:08.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_1.status 2024-09-14T05:53:08.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:53:08.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:08.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-14T05:53:08.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_1.status 2024-09-14T05:53:08.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-09-14T05:53:08.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:53:08.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:53:08.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:53:08.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:53:08.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:53:08.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:53:08.645 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:53:08.645 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 86 = '' 2024-09-14T05:53:08.645 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:53:08.645 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-14T05:53:10.247 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_1.status 2024-09-14T05:53:10.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:53:10.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:10.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-14T05:53:10.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_1.status 2024-09-14T05:53:10.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-09-14T05:53:10.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:53:10.302 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:53:10.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=86 2024-09-14T05:53:10.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 86 = 86 2024-09-14T05:53:10.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:53:10.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_1 snap 2024-09-14T05:53:10.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:53:10.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:10.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-14T05:53:10.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:53:10.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:53:10.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:53:10.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:53:11.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_1@snap 2024-09-14T05:53:11.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_1': 2024-09-14T05:53:11.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:53:11.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:53:11.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-14T05:53:11.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c1ab8ac170 2024-09-14T05:53:11.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c1ab8ac170 2024-09-14T05:53:11.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:53:11.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:53:11.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:53:11.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:53:11.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:44:33 2024 2024-09-14T05:53:11.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:44:33 2024 2024-09-14T05:53:11.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:44:33 2024 2024-09-14T05:53:11.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:53:11.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:53:11.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:53:11.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: cc05da34-933d-4ab7-a8cd-8e584cdd7250 2024-09-14T05:53:11.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:53:11.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:53:11.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_1 snap 2024-09-14T05:53:11.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:11.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-14T05:53:11.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:53:11.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:53:11.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_1.export 2024-09-14T05:53:11.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_1.export 2024-09-14T05:53:11.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_1.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_1.export 2024-09-14T05:53:11.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_1@snap /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_1.export 2024-09-14T05:53:11.532 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:53:11.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_1@snap /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_1.export 2024-09-14T05:53:11.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:53:11.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_1.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_1.export 2024-09-14T05:53:11.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_1.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_1.export 2024-09-14T05:53:11.836 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:53:11.836 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:53:11.836 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_2 2024-09-14T05:53:11.836 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_2 snap 2024-09-14T05:53:11.836 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:53:11.836 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:11.836 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-14T05:53:11.836 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:53:11.836 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_2@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T05:53:13.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_2 2024-09-14T05:53:13.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:53:13.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:13.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-14T05:53:13.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_2 started 2024-09-14T05:53:13.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:53:13.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:13.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-14T05:53:13.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:53:13.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:53:13.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:53:13.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:53:14.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_2 started 2024-09-14T05:53:14.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:53:14.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:14.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-14T05:53:14.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:53:14.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:53:14.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:53:14.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_2 2024-09-14T05:53:14.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:53:14.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:53:14.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:53:14.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:53:14.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:53:14.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:53:14.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:53:14.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:53:14.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:53:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:53:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:53:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_2 2024-09-14T05:53:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:53:14.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:53:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:53:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:53:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:53:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:53:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:53:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:53:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:53:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:53:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:53:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:53:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:53:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:53:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:53:14.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:53:14.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:53:14.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:53:14.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:53:14.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:53:14.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:53:14.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:53:14.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:53:14.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:53:14.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:53:14.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:53:14.112 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:53:14.112 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:53:14.112 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:53:14.112 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:53:14.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:53:14.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:53:14.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:53:14.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_2 2024-09-14T05:53:14.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_2'] 2024-09-14T05:53:14.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:53:14.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:53:14.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:53:14.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_2 2024-09-14T05:53:14.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-14T05:53:14.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:53:14.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:53:14.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T05:53:14.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-14T05:53:14.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:53:14.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:53:14.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:53:14.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-14T05:53:14.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T05:53:14.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:53:14.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:53:14.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-14T05:53:14.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:53:14.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T05:53:14.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-14T05:53:14.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:53:14.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:53:14.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T05:53:14.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-14T05:53:14.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:53:14.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:53:14.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:53:14.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:53:14.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T05:53:14.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:53:14.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:53:14.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-14T05:53:14.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-14T05:53:14.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T05:53:14.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:53:14.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:53:14.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:53:14.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_2 2024-09-14T05:53:14.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_2'] 2024-09-14T05:53:14.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:53:14.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:53:14.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:53:14.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_2 2024-09-14T05:53:14.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-14T05:53:14.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:53:14.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:53:14.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T05:53:14.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-14T05:53:14.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:53:14.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:53:14.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:53:14.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-14T05:53:14.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T05:53:14.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:53:14.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:53:14.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-14T05:53:14.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:53:14.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T05:53:14.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-14T05:53:14.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:53:14.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:53:14.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T05:53:14.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-14T05:53:14.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:53:14.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:53:14.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:53:14.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:53:14.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T05:53:14.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:53:14.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:53:14.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-14T05:53:14.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-14T05:53:14.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T05:53:14.327 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:53:14.327 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:53:14.327 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:53:14.327 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_2 2024-09-14T05:53:14.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:53:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:53:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:53:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:53:14.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:53:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:53:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:53:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:53:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_2 2024-09-14T05:53:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:53:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:53:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-14T05:53:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:53:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:53:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_2 2024-09-14T05:53:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:53:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:53:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:14.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-14T05:53:14.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_2.status 2024-09-14T05:53:14.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_2.status 2024-09-14T05:53:14.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:53:14.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_2.status 2024-09-14T05:53:14.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_2.status 2024-09-14T05:53:14.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_2.status 2024-09-14T05:53:14.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:53:14.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_2.status 2024-09-14T05:53:14.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_2 2024-09-14T05:53:14.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:53:14.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:14.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-14T05:53:14.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_2 2024-09-14T05:53:15.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 88 2024-09-14T05:53:15.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_2 /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_2.status 2024-09-14T05:53:15.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:53:15.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:15.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-14T05:53:15.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_2.status 2024-09-14T05:53:15.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_2 --format xml 2024-09-14T05:53:15.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:53:15.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:53:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=88 2024-09-14T05:53:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:53:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:53:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:53:15.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_2.status 2024-09-14T05:53:15.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:53:15.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:15.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-14T05:53:15.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_2.status 2024-09-14T05:53:15.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-09-14T05:53:15.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:53:15.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:53:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=36 2024-09-14T05:53:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 88 = 36 2024-09-14T05:53:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:53:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:53:15.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_2.status 2024-09-14T05:53:15.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:53:15.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:15.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-14T05:53:15.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_2.status 2024-09-14T05:53:15.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-09-14T05:53:15.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:53:15.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:53:15.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=36 2024-09-14T05:53:15.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 88 = 36 2024-09-14T05:53:15.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:53:15.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:53:16.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_2.status 2024-09-14T05:53:16.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:53:16.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:16.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-14T05:53:16.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_2.status 2024-09-14T05:53:16.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-09-14T05:53:16.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:53:16.722 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:53:16.722 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:53:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:53:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:53:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:53:16.724 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:53:16.724 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 88 = '' 2024-09-14T05:53:16.724 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:53:16.724 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-14T05:53:18.325 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_2.status 2024-09-14T05:53:18.325 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:53:18.325 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:18.325 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-14T05:53:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_2.status 2024-09-14T05:53:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-09-14T05:53:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:53:18.421 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:53:18.423 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=88 2024-09-14T05:53:18.423 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 88 = 88 2024-09-14T05:53:18.423 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:53:18.423 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_2 snap 2024-09-14T05:53:18.423 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:53:18.423 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:18.423 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-14T05:53:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:53:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:53:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:53:18.424 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:53:19.424 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_2@snap 2024-09-14T05:53:19.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_2': 2024-09-14T05:53:19.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:53:19.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:53:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-14T05:53:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c192707173 2024-09-14T05:53:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c192707173 2024-09-14T05:53:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:53:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:53:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:53:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:53:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:44:47 2024 2024-09-14T05:53:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:44:47 2024 2024-09-14T05:53:19.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:44:47 2024 2024-09-14T05:53:19.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:53:19.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:53:19.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:53:19.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: a89e8edb-c81f-4b3c-b444-29c2fea9908b 2024-09-14T05:53:19.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:53:19.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:53:19.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_2 snap 2024-09-14T05:53:19.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:19.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-14T05:53:19.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:53:19.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:53:19.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_2.export 2024-09-14T05:53:19.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_2.export 2024-09-14T05:53:19.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_2.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_2.export 2024-09-14T05:53:19.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_2@snap /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_2.export 2024-09-14T05:53:19.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:53:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_2@snap /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_2.export 2024-09-14T05:53:19.852 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:53:19.859 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_2.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_2.export 2024-09-14T05:53:19.913 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_2.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_2.export 2024-09-14T05:53:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:53:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:53:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_3 2024-09-14T05:53:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_3 snap 2024-09-14T05:53:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:53:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-14T05:53:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:53:19.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_3@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T05:53:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_3 2024-09-14T05:53:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:53:21.158 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:21.158 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-14T05:53:21.158 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_3 started 2024-09-14T05:53:21.158 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:53:21.158 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:21.158 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-14T05:53:21.158 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:53:21.158 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:53:21.158 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:53:21.158 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:53:22.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_3 started 2024-09-14T05:53:22.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:53:22.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:22.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-14T05:53:22.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:53:22.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:53:22.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:53:22.160 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_3 2024-09-14T05:53:22.160 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:53:22.161 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:53:22.161 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:53:22.161 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:53:22.161 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:53:22.161 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:53:22.161 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:53:22.161 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:53:22.161 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:53:22.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:53:22.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:53:22.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_3 2024-09-14T05:53:22.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:53:22.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:53:22.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:53:22.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:53:22.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:53:22.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:53:22.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:53:22.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:53:22.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:53:22.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:53:22.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:53:22.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:53:22.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:53:22.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:53:22.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:53:22.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:53:22.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:53:22.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:53:22.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:53:22.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:53:22.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:53:22.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:53:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:53:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:53:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:53:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:53:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:53:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:53:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:53:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:53:22.186 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:53:22.186 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:53:22.186 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:53:22.186 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_3 2024-09-14T05:53:22.264 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_3'] 2024-09-14T05:53:22.264 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:53:22.264 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:53:22.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:53:22.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_3 2024-09-14T05:53:22.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-14T05:53:22.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:53:22.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:53:22.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T05:53:22.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-14T05:53:22.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:53:22.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:53:22.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:53:22.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-14T05:53:22.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T05:53:22.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:53:22.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:53:22.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-14T05:53:22.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:53:22.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T05:53:22.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-14T05:53:22.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:53:22.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:53:22.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T05:53:22.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-14T05:53:22.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:53:22.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:53:22.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:53:22.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:53:22.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T05:53:22.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:53:22.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:53:22.270 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-14T05:53:22.270 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-14T05:53:22.270 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T05:53:22.286 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:53:22.286 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:53:22.286 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:53:22.286 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_3 2024-09-14T05:53:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_3'] 2024-09-14T05:53:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:53:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:53:22.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:53:22.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_3 2024-09-14T05:53:22.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-14T05:53:22.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:53:22.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:53:22.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T05:53:22.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-14T05:53:22.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:53:22.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:53:22.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:53:22.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-14T05:53:22.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T05:53:22.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:53:22.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:53:22.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-14T05:53:22.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:53:22.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T05:53:22.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-14T05:53:22.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:53:22.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:53:22.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T05:53:22.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-14T05:53:22.363 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:53:22.363 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:53:22.363 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:53:22.363 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:53:22.363 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T05:53:22.363 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:53:22.363 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:53:22.363 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-14T05:53:22.363 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-14T05:53:22.363 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T05:53:22.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:53:22.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:53:22.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:53:22.380 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_3 2024-09-14T05:53:22.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:53:22.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:53:22.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:53:22.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:53:22.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:53:22.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:53:22.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:53:22.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:53:22.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_3 2024-09-14T05:53:22.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:53:22.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:53:22.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:22.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-14T05:53:22.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:53:22.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:53:22.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_3 2024-09-14T05:53:22.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:53:22.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:53:22.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:22.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-14T05:53:22.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_3.status 2024-09-14T05:53:22.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_3.status 2024-09-14T05:53:22.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:53:22.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_3.status 2024-09-14T05:53:22.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_3.status 2024-09-14T05:53:22.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_3.status 2024-09-14T05:53:22.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:53:22.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_3.status 2024-09-14T05:53:22.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_3 2024-09-14T05:53:22.498 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:53:22.498 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:22.498 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-14T05:53:22.498 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_3 2024-09-14T05:53:23.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 90 2024-09-14T05:53:23.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_3 /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_3.status 2024-09-14T05:53:23.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:53:23.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:23.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-14T05:53:23.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_3.status 2024-09-14T05:53:23.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_3 --format xml 2024-09-14T05:53:23.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:53:23.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:53:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=90 2024-09-14T05:53:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:53:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:53:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:53:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_3.status 2024-09-14T05:53:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:53:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-14T05:53:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_3.status 2024-09-14T05:53:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-09-14T05:53:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:53:23.435 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:53:23.436 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=37 2024-09-14T05:53:23.437 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 90 = 37 2024-09-14T05:53:23.437 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:53:23.437 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:53:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_3.status 2024-09-14T05:53:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:53:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-14T05:53:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_3.status 2024-09-14T05:53:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-09-14T05:53:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:53:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:53:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=37 2024-09-14T05:53:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 90 = 37 2024-09-14T05:53:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:53:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:53:24.691 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_3.status 2024-09-14T05:53:24.691 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:53:24.691 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:24.691 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-14T05:53:24.691 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_3.status 2024-09-14T05:53:24.691 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-09-14T05:53:24.691 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:53:24.741 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:53:24.741 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:53:24.742 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:53:24.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:53:24.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:53:24.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:53:24.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 90 = '' 2024-09-14T05:53:24.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:53:24.744 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-14T05:53:26.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_3.status 2024-09-14T05:53:26.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:53:26.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:26.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-14T05:53:26.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_3.status 2024-09-14T05:53:26.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-09-14T05:53:26.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:53:26.401 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:53:26.403 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=90 2024-09-14T05:53:26.403 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 90 = 90 2024-09-14T05:53:26.403 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:53:26.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_3 snap 2024-09-14T05:53:26.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:53:26.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:26.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-14T05:53:26.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:53:26.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:53:26.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:53:26.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:53:27.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_3@snap 2024-09-14T05:53:27.455 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_3': 2024-09-14T05:53:27.455 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:53:27.455 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:53:27.455 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-14T05:53:27.455 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c11677250e 2024-09-14T05:53:27.455 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c11677250e 2024-09-14T05:53:27.455 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:53:27.455 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:53:27.455 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:53:27.456 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:53:27.456 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:44:56 2024 2024-09-14T05:53:27.456 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:44:56 2024 2024-09-14T05:53:27.456 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:44:56 2024 2024-09-14T05:53:27.456 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:53:27.456 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:53:27.456 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:53:27.456 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 847da547-970e-41a0-bee0-110873986de0 2024-09-14T05:53:27.456 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:53:27.459 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:53:27.459 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_3 snap 2024-09-14T05:53:27.459 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:27.459 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-14T05:53:27.459 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:53:27.459 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:53:27.459 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_3.export 2024-09-14T05:53:27.459 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_3.export 2024-09-14T05:53:27.459 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_3.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_3.export 2024-09-14T05:53:27.460 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_3@snap /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_3.export 2024-09-14T05:53:27.621 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:53:27.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_3@snap /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_3.export 2024-09-14T05:53:27.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:53:27.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_3.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_3.export 2024-09-14T05:53:27.914 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_3.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_3.export 2024-09-14T05:53:27.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:53:27.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:53:27.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_4 2024-09-14T05:53:27.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_4 snap 2024-09-14T05:53:27.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:53:27.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:27.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-14T05:53:27.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:53:27.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_4@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T05:53:29.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_4 2024-09-14T05:53:29.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:53:29.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:29.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-14T05:53:29.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_4 started 2024-09-14T05:53:29.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:53:29.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:29.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-14T05:53:29.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:53:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:53:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:53:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:53:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_4 started 2024-09-14T05:53:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:53:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-14T05:53:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:53:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:53:30.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:53:30.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_4 2024-09-14T05:53:30.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:53:30.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:53:30.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:53:30.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:53:30.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:53:30.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:53:30.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:53:30.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:53:30.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:53:30.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:53:30.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:53:30.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2024-09-14T05:53:30.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:53:30.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:53:30.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:53:30.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:53:30.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:53:30.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:53:30.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:53:30.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:53:30.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:53:30.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:53:30.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:53:30.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:53:30.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:53:30.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:53:30.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:53:30.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:53:30.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:53:30.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:53:30.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:53:30.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:53:30.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:53:30.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:53:30.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:53:30.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:53:30.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:53:30.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:53:30.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:53:30.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:53:30.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:53:30.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:53:30.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:53:30.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:53:30.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:53:30.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2024-09-14T05:53:30.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_4'] 2024-09-14T05:53:30.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:53:30.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:53:30.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:53:30.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_4 2024-09-14T05:53:30.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-14T05:53:30.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:53:30.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:53:30.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T05:53:30.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-14T05:53:30.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:53:30.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:53:30.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:53:30.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-14T05:53:30.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T05:53:30.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:53:30.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:53:30.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-14T05:53:30.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:53:30.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T05:53:30.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-14T05:53:30.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:53:30.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:53:30.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T05:53:30.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-14T05:53:30.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:53:30.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:53:30.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:53:30.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:53:30.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T05:53:30.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:53:30.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:53:30.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-14T05:53:30.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-14T05:53:30.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T05:53:30.272 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:53:30.272 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:53:30.272 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:53:30.272 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_4 2024-09-14T05:53:30.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_4'] 2024-09-14T05:53:30.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:53:30.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:53:30.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:53:30.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_4 2024-09-14T05:53:30.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-14T05:53:30.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:53:30.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:53:30.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T05:53:30.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-14T05:53:30.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:53:30.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:53:30.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:53:30.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-14T05:53:30.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T05:53:30.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:53:30.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:53:30.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-14T05:53:30.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:53:30.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T05:53:30.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-14T05:53:30.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:53:30.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:53:30.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T05:53:30.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-14T05:53:30.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:53:30.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:53:30.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:53:30.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:53:30.350 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T05:53:30.350 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:53:30.350 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:53:30.350 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-14T05:53:30.350 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-14T05:53:30.350 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T05:53:30.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:53:30.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:53:30.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:53:30.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_4 2024-09-14T05:53:30.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:53:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:53:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:53:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:53:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:53:30.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:53:30.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:53:30.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:53:30.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_4 2024-09-14T05:53:30.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:53:30.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:53:30.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:30.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-14T05:53:30.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:53:30.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:53:30.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_4 2024-09-14T05:53:30.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:53:30.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:53:30.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:30.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-14T05:53:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_4.status 2024-09-14T05:53:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_4.status 2024-09-14T05:53:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:53:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_4.status 2024-09-14T05:53:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_4.status 2024-09-14T05:53:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_4.status 2024-09-14T05:53:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:53:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_4.status 2024-09-14T05:53:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_4 2024-09-14T05:53:30.489 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:53:30.489 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:30.489 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-14T05:53:30.489 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_4 2024-09-14T05:53:31.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 92 2024-09-14T05:53:31.160 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_4 /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_4.status 2024-09-14T05:53:31.160 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:53:31.160 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:31.160 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-14T05:53:31.161 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_4.status 2024-09-14T05:53:31.161 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_4 --format xml 2024-09-14T05:53:31.161 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:53:31.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:53:31.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=92 2024-09-14T05:53:31.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:53:31.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:53:31.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:53:31.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_4.status 2024-09-14T05:53:31.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:53:31.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:31.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-14T05:53:31.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_4.status 2024-09-14T05:53:31.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-09-14T05:53:31.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:53:31.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:53:31.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=38 2024-09-14T05:53:31.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 92 = 38 2024-09-14T05:53:31.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:53:31.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:53:31.864 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_4.status 2024-09-14T05:53:31.865 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:53:31.865 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:31.865 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-14T05:53:31.865 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_4.status 2024-09-14T05:53:31.865 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-09-14T05:53:31.865 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:53:31.913 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:53:31.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=38 2024-09-14T05:53:31.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 92 = 38 2024-09-14T05:53:31.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:53:31.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:53:32.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_4.status 2024-09-14T05:53:32.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:53:32.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:32.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-14T05:53:32.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_4.status 2024-09-14T05:53:32.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-09-14T05:53:32.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:53:32.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:53:32.773 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:53:32.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:53:32.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:53:32.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:53:32.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:53:32.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 92 = '' 2024-09-14T05:53:32.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:53:32.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-14T05:53:34.376 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_4.status 2024-09-14T05:53:34.376 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:53:34.377 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:34.377 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-14T05:53:34.377 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_4.status 2024-09-14T05:53:34.377 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-09-14T05:53:34.377 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:53:34.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:53:34.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=92 2024-09-14T05:53:34.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 92 = 92 2024-09-14T05:53:34.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:53:34.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_4 snap 2024-09-14T05:53:34.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:53:34.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:34.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-14T05:53:34.435 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:53:34.435 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:53:34.435 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:53:34.435 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:53:35.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_4@snap 2024-09-14T05:53:35.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_4': 2024-09-14T05:53:35.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:53:35.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:53:35.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-14T05:53:35.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c1ab05bc3a 2024-09-14T05:53:35.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c1ab05bc3a 2024-09-14T05:53:35.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:53:35.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:53:35.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:53:35.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:53:35.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:45:02 2024 2024-09-14T05:53:35.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:45:02 2024 2024-09-14T05:53:35.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:45:02 2024 2024-09-14T05:53:35.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:53:35.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:53:35.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:53:35.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 476c0897-664e-4ed7-a165-44f4a9d2c1a5 2024-09-14T05:53:35.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:53:35.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:53:35.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_4 snap 2024-09-14T05:53:35.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:35.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-14T05:53:35.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:53:35.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:53:35.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_4.export 2024-09-14T05:53:35.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_4.export 2024-09-14T05:53:35.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_4.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_4.export 2024-09-14T05:53:35.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_4@snap /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_4.export 2024-09-14T05:53:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:53:35.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_4@snap /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_4.export 2024-09-14T05:53:35.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:53:35.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_4.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_4.export 2024-09-14T05:53:35.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_4.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_4.export 2024-09-14T05:53:35.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:53:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:53:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_5 2024-09-14T05:53:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_5 snap 2024-09-14T05:53:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:53:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-14T05:53:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:53:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_5@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T05:53:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_5 2024-09-14T05:53:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:53:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-14T05:53:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_5 started 2024-09-14T05:53:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:53:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-14T05:53:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:53:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:53:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:53:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:53:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_5 started 2024-09-14T05:53:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:53:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-14T05:53:38.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:53:38.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:53:38.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:53:38.171 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_5 2024-09-14T05:53:38.171 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:53:38.171 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:53:38.171 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:53:38.171 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:53:38.172 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:53:38.172 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:53:38.172 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:53:38.172 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:53:38.172 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:53:38.172 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:53:38.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:53:38.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2024-09-14T05:53:38.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:53:38.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:53:38.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:53:38.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:53:38.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:53:38.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:53:38.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:53:38.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:53:38.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:53:38.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:53:38.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:53:38.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:53:38.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:53:38.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:53:38.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:53:38.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:53:38.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:53:38.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:53:38.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:53:38.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:53:38.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:53:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:53:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:53:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:53:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:53:38.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:53:38.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:53:38.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:53:38.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:53:38.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:53:38.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:53:38.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:53:38.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:53:38.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2024-09-14T05:53:38.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-09-14T05:53:38.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:53:38.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:53:38.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:53:38.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_5 2024-09-14T05:53:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-14T05:53:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:53:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:53:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T05:53:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-14T05:53:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:53:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:53:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:53:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-14T05:53:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T05:53:38.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:53:38.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:53:38.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-14T05:53:38.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:53:38.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T05:53:38.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-14T05:53:38.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:53:38.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:53:38.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T05:53:38.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-14T05:53:38.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:53:38.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:53:38.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:53:38.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:53:38.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T05:53:38.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:53:38.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:53:38.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-14T05:53:38.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-14T05:53:38.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T05:53:38.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:53:38.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:53:38.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:53:38.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_5 2024-09-14T05:53:38.371 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-09-14T05:53:38.371 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:53:38.371 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:53:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:53:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_5 2024-09-14T05:53:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-14T05:53:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:53:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:53:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T05:53:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-14T05:53:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:53:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:53:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:53:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-14T05:53:38.375 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T05:53:38.375 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:53:38.375 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:53:38.375 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-14T05:53:38.375 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:53:38.375 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T05:53:38.375 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-14T05:53:38.375 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:53:38.375 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:53:38.375 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T05:53:38.375 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-14T05:53:38.375 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:53:38.375 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:53:38.376 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:53:38.376 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:53:38.376 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T05:53:38.376 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:53:38.376 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:53:38.376 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-14T05:53:38.376 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-14T05:53:38.376 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T05:53:38.392 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:53:38.392 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:53:38.392 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:53:38.392 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_5 2024-09-14T05:53:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:53:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:53:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:53:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:53:38.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:53:38.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:53:38.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:53:38.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:53:38.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_5 2024-09-14T05:53:38.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:53:38.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:53:38.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:38.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-14T05:53:38.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:53:38.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:53:38.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_5 2024-09-14T05:53:38.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:53:38.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:53:38.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:38.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-14T05:53:38.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_5.status 2024-09-14T05:53:38.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_5.status 2024-09-14T05:53:38.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:53:38.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_5.status 2024-09-14T05:53:38.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_5.status 2024-09-14T05:53:38.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_5.status 2024-09-14T05:53:38.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:53:38.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_5.status 2024-09-14T05:53:38.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_5 2024-09-14T05:53:38.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:53:38.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:38.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-14T05:53:38.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_5 2024-09-14T05:53:39.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 94 2024-09-14T05:53:39.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_5 /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_5.status 2024-09-14T05:53:39.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:53:39.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:39.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-14T05:53:39.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_5.status 2024-09-14T05:53:39.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_5 --format xml 2024-09-14T05:53:39.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:53:39.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:53:39.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=94 2024-09-14T05:53:39.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:53:39.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:53:39.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:53:39.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_5.status 2024-09-14T05:53:39.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:53:39.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:39.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-14T05:53:39.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_5.status 2024-09-14T05:53:39.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-09-14T05:53:39.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:53:39.498 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:53:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=39 2024-09-14T05:53:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 94 = 39 2024-09-14T05:53:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:53:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:53:39.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_5.status 2024-09-14T05:53:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:53:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-14T05:53:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_5.status 2024-09-14T05:53:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-09-14T05:53:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:53:39.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:53:39.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=39 2024-09-14T05:53:39.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 94 = 39 2024-09-14T05:53:39.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:53:39.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:53:40.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_5.status 2024-09-14T05:53:40.758 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:53:40.758 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:40.758 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-14T05:53:40.758 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_5.status 2024-09-14T05:53:40.758 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-09-14T05:53:40.758 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:53:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:53:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:53:40.812 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:53:40.812 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:53:40.812 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:53:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:53:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 94 = '' 2024-09-14T05:53:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:53:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-14T05:53:41.488 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8310349483654986) > weight (0.25) 2024-09-14T05:53:41.488 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-09-14T05:53:41.488 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-09-14T05:53:41.489 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-09-14T05:53:41.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:2024-09-14T05:53:41.488+0000 7f6a0ca17640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 49252) UID: 1000 2024-09-14T05:53:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_5.status 2024-09-14T05:53:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:53:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-14T05:53:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_5.status 2024-09-14T05:53:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-09-14T05:53:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:53:42.468 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:53:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=94 2024-09-14T05:53:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 94 = 94 2024-09-14T05:53:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:53:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_5 snap 2024-09-14T05:53:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:53:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-14T05:53:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:53:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:53:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:53:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:53:42.490 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-09-14T05:53:42.490 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T05:53:42.490 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-09-14T05:53:42.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-14T05:53:42.491 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-14T05:53:42.493 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-14T05:53:42.493 INFO:tasks.rbd_mirror_thrash:waiting for 86 secs before thrashing 2024-09-14T05:53:43.473 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_5@snap 2024-09-14T05:53:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_5': 2024-09-14T05:53:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:53:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:53:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-14T05:53:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c14342ae5d 2024-09-14T05:53:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c14342ae5d 2024-09-14T05:53:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:53:43.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:53:43.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:53:43.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:53:43.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:45:13 2024 2024-09-14T05:53:43.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:45:13 2024 2024-09-14T05:53:43.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:45:13 2024 2024-09-14T05:53:43.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:53:43.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:53:43.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:53:43.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 5962be87-a0c9-4726-93e1-f3aad0bd9173 2024-09-14T05:53:43.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:53:43.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:53:43.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_5 snap 2024-09-14T05:53:43.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:43.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-14T05:53:43.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:53:43.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:53:43.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_5.export 2024-09-14T05:53:43.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_5.export 2024-09-14T05:53:43.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_5.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_5.export 2024-09-14T05:53:43.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_5@snap /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_5.export 2024-09-14T05:53:43.688 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:53:43.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_5@snap /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_5.export 2024-09-14T05:53:43.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:53:43.926 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_5.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_5.export 2024-09-14T05:53:43.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_5.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_5.export 2024-09-14T05:53:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:53:44.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:53:44.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_6 2024-09-14T05:53:44.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_6 snap 2024-09-14T05:53:44.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:53:44.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:44.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-14T05:53:44.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:53:44.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_6@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T05:53:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_6 2024-09-14T05:53:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:53:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-14T05:53:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_6 started 2024-09-14T05:53:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:53:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-14T05:53:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:53:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:53:45.193 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:53:45.193 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:53:46.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_6 started 2024-09-14T05:53:46.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:53:46.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:46.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-14T05:53:46.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:53:46.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:53:46.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:53:46.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_6 2024-09-14T05:53:46.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:53:46.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:53:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:53:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:53:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:53:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:53:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:53:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:53:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:53:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:53:46.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:53:46.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_6 2024-09-14T05:53:46.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:53:46.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:53:46.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:53:46.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:53:46.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:53:46.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:53:46.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:53:46.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:53:46.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:53:46.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:53:46.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:53:46.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:53:46.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:53:46.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:53:46.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:53:46.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:53:46.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:53:46.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:53:46.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:53:46.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:53:46.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:53:46.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:53:46.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:53:46.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:53:46.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:53:46.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:53:46.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:53:46.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:53:46.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:53:46.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:53:46.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:53:46.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:53:46.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:53:46.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_6 2024-09-14T05:53:46.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2024-09-14T05:53:46.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:53:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:53:46.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:53:46.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_6 2024-09-14T05:53:46.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-14T05:53:46.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:53:46.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:53:46.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T05:53:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-14T05:53:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:53:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:53:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:53:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-14T05:53:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T05:53:46.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:53:46.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:53:46.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-14T05:53:46.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:53:46.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T05:53:46.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-14T05:53:46.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:53:46.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:53:46.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T05:53:46.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-14T05:53:46.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:53:46.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:53:46.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:53:46.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:53:46.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T05:53:46.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:53:46.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:53:46.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-14T05:53:46.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-14T05:53:46.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T05:53:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:53:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:53:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:53:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_6 2024-09-14T05:53:46.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2024-09-14T05:53:46.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:53:46.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:53:46.399 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:53:46.399 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_6 2024-09-14T05:53:46.399 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-14T05:53:46.399 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:53:46.400 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:53:46.400 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T05:53:46.400 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-14T05:53:46.400 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:53:46.400 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:53:46.400 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:53:46.401 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-14T05:53:46.401 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T05:53:46.401 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:53:46.401 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:53:46.401 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-14T05:53:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:53:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T05:53:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-14T05:53:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:53:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:53:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T05:53:46.403 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-14T05:53:46.403 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:53:46.403 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:53:46.403 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:53:46.403 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:53:46.403 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T05:53:46.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:53:46.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:53:46.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-14T05:53:46.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-14T05:53:46.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T05:53:46.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:53:46.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:53:46.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:53:46.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_6 2024-09-14T05:53:46.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:53:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:53:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:53:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:53:46.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:53:46.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:53:46.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:53:46.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:53:46.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_6 2024-09-14T05:53:46.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:53:46.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:53:46.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:46.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-14T05:53:46.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:53:46.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:53:46.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_6 2024-09-14T05:53:46.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:53:46.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:53:46.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:46.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-14T05:53:46.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_6.status 2024-09-14T05:53:46.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_6.status 2024-09-14T05:53:46.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:53:46.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_6.status 2024-09-14T05:53:46.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_6.status 2024-09-14T05:53:46.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_6.status 2024-09-14T05:53:46.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:53:46.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_6.status 2024-09-14T05:53:46.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_6 2024-09-14T05:53:46.538 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:53:46.538 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:46.538 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-14T05:53:46.538 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_6 2024-09-14T05:53:47.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 96 2024-09-14T05:53:47.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_6 /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_6.status 2024-09-14T05:53:47.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:53:47.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:47.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-14T05:53:47.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_6.status 2024-09-14T05:53:47.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_6 --format xml 2024-09-14T05:53:47.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:53:47.251 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:53:47.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=96 2024-09-14T05:53:47.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:53:47.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:53:47.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:53:47.454 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_6.status 2024-09-14T05:53:47.454 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:53:47.454 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:47.455 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-14T05:53:47.455 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_6.status 2024-09-14T05:53:47.455 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-09-14T05:53:47.455 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:53:47.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:53:47.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=40 2024-09-14T05:53:47.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 96 = 40 2024-09-14T05:53:47.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:53:47.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:53:47.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_6.status 2024-09-14T05:53:47.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:53:47.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:47.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-14T05:53:47.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_6.status 2024-09-14T05:53:47.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-09-14T05:53:47.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:53:47.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:53:47.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:53:47.960 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:53:47.960 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:53:47.960 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:53:47.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:53:47.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 96 = '' 2024-09-14T05:53:47.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:53:47.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:53:48.762 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_6.status 2024-09-14T05:53:48.762 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:53:48.762 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:48.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-14T05:53:48.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_6.status 2024-09-14T05:53:48.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-09-14T05:53:48.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:53:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:53:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:53:48.816 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:53:48.816 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:53:48.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:53:48.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:53:48.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 96 = '' 2024-09-14T05:53:48.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:53:48.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-14T05:53:50.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_6.status 2024-09-14T05:53:50.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:53:50.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:50.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-14T05:53:50.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_6.status 2024-09-14T05:53:50.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-09-14T05:53:50.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:53:50.472 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:53:50.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=96 2024-09-14T05:53:50.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 96 = 96 2024-09-14T05:53:50.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:53:50.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_6 snap 2024-09-14T05:53:50.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:53:50.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:50.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-14T05:53:50.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:53:50.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:53:50.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:53:50.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:53:51.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_6@snap 2024-09-14T05:53:51.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_6': 2024-09-14T05:53:51.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:53:51.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:53:51.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-14T05:53:51.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c1badc0d69 2024-09-14T05:53:51.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c1badc0d69 2024-09-14T05:53:51.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:53:51.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:53:51.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:53:51.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:53:51.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:45:23 2024 2024-09-14T05:53:51.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:45:23 2024 2024-09-14T05:53:51.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:45:23 2024 2024-09-14T05:53:51.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:53:51.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:53:51.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:53:51.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: a86da690-7adb-435f-a4fb-fb58a8d79cb7 2024-09-14T05:53:51.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:53:51.529 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:53:51.529 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_6 snap 2024-09-14T05:53:51.529 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:51.529 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-14T05:53:51.529 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:53:51.529 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:53:51.529 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_6.export 2024-09-14T05:53:51.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_6.export 2024-09-14T05:53:51.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_6.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_6.export 2024-09-14T05:53:51.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_6@snap /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_6.export 2024-09-14T05:53:51.698 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:53:51.704 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_6@snap /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_6.export 2024-09-14T05:53:51.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:53:51.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_6.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_6.export 2024-09-14T05:53:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_6.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_6.export 2024-09-14T05:53:52.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:53:52.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:53:52.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_7 2024-09-14T05:53:52.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_7 snap 2024-09-14T05:53:52.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:53:52.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:52.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-14T05:53:52.010 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:53:52.011 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_7@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T05:53:53.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_7 2024-09-14T05:53:53.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:53:53.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:53.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-14T05:53:53.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_7 started 2024-09-14T05:53:53.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:53:53.770 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:53.770 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-14T05:53:53.770 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:53:53.770 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:53:53.770 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:53:53.770 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:53:54.770 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_7 started 2024-09-14T05:53:54.770 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:53:54.770 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:54.771 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-14T05:53:54.771 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:53:54.771 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:53:54.771 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:53:54.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_7 2024-09-14T05:53:54.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:53:54.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:53:54.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:53:54.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:53:54.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:53:54.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:53:54.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:53:54.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:53:54.773 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:53:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:53:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:53:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_7 2024-09-14T05:53:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:53:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:53:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:53:54.776 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:53:54.776 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:53:54.776 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:53:54.776 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:53:54.776 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:53:54.776 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:53:54.776 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:53:54.776 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:53:54.776 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:53:54.776 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:53:54.776 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:53:54.776 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:53:54.776 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:53:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:53:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:53:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:53:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:53:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:53:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:53:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:53:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:53:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:53:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:53:54.778 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:53:54.778 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:53:54.778 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:53:54.778 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:53:54.797 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:53:54.797 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:53:54.797 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:53:54.797 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_7 2024-09-14T05:53:54.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2024-09-14T05:53:54.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:53:54.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:53:54.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:53:54.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_7 2024-09-14T05:53:54.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-14T05:53:54.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:53:54.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:53:54.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T05:53:54.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-14T05:53:54.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:53:54.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:53:54.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:53:54.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-14T05:53:54.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T05:53:54.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:53:54.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:53:54.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-14T05:53:54.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:53:54.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T05:53:54.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-14T05:53:54.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:53:54.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:53:54.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T05:53:54.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-14T05:53:54.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:53:54.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:53:54.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:53:54.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:53:54.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T05:53:54.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:53:54.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:53:54.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-14T05:53:54.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-14T05:53:54.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T05:53:54.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:53:54.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:53:54.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:53:54.899 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_7 2024-09-14T05:53:54.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2024-09-14T05:53:54.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:53:54.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:53:54.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:53:54.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_7 2024-09-14T05:53:54.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-14T05:53:54.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:53:54.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:53:54.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T05:53:54.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-14T05:53:54.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:53:54.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:53:54.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:53:54.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-14T05:53:54.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T05:53:54.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:53:54.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:53:54.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-14T05:53:54.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:53:54.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T05:53:54.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-14T05:53:54.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:53:54.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:53:54.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T05:53:54.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-14T05:53:54.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:53:54.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:53:54.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:53:54.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:53:54.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T05:53:54.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:53:54.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:53:54.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-14T05:53:54.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-14T05:53:54.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T05:53:54.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:53:54.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:53:54.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:53:54.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_7 2024-09-14T05:53:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:53:55.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:53:55.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:53:55.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:53:55.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:53:55.112 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:53:55.112 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:53:55.112 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:53:55.113 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_7 2024-09-14T05:53:55.113 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:53:55.113 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:53:55.113 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:55.113 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-14T05:53:55.113 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:53:55.113 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:53:55.113 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_7 2024-09-14T05:53:55.113 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:53:55.113 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:53:55.113 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:55.113 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-14T05:53:55.113 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_7.status 2024-09-14T05:53:55.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_7.status 2024-09-14T05:53:55.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:53:55.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_7.status 2024-09-14T05:53:55.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_7.status 2024-09-14T05:53:55.114 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_7.status 2024-09-14T05:53:55.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:53:55.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_7.status 2024-09-14T05:53:55.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_7 2024-09-14T05:53:55.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:53:55.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:55.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-14T05:53:55.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_7 2024-09-14T05:53:55.768 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 98 2024-09-14T05:53:55.779 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_7 /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_7.status 2024-09-14T05:53:55.779 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:53:55.779 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:55.779 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-14T05:53:55.779 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_7.status 2024-09-14T05:53:55.779 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_7 --format xml 2024-09-14T05:53:55.779 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:53:55.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:53:55.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=98 2024-09-14T05:53:55.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:53:55.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:53:55.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:53:56.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_7.status 2024-09-14T05:53:56.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:53:56.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:56.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-14T05:53:56.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_7.status 2024-09-14T05:53:56.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-09-14T05:53:56.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:53:56.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:53:56.076 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:53:56.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:53:56.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:53:56.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:53:56.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:53:56.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 98 = '' 2024-09-14T05:53:56.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:53:56.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:53:56.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_7.status 2024-09-14T05:53:56.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:53:56.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:56.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-14T05:53:56.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_7.status 2024-09-14T05:53:56.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-09-14T05:53:56.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:53:56.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:53:56.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:53:56.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:53:56.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:53:56.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:53:56.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:53:56.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 98 = '' 2024-09-14T05:53:56.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:53:56.534 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:53:57.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_7.status 2024-09-14T05:53:57.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:53:57.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:57.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-14T05:53:57.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_7.status 2024-09-14T05:53:57.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-09-14T05:53:57.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:53:57.380 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:53:57.382 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=98 2024-09-14T05:53:57.382 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 98 = 98 2024-09-14T05:53:57.382 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:53:57.383 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_7 snap 2024-09-14T05:53:57.383 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:53:57.383 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:57.383 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-14T05:53:57.383 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:53:57.383 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:53:57.383 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:53:57.383 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:53:58.383 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_7@snap 2024-09-14T05:53:58.426 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_7': 2024-09-14T05:53:58.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:53:58.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:53:58.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-14T05:53:58.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c1783f46b3 2024-09-14T05:53:58.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c1783f46b3 2024-09-14T05:53:58.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:53:58.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:53:58.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:53:58.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:53:58.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:45:29 2024 2024-09-14T05:53:58.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:45:29 2024 2024-09-14T05:53:58.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:45:29 2024 2024-09-14T05:53:58.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:53:58.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:53:58.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:53:58.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 34d473d8-f5ad-4be8-be0a-057254b091f7 2024-09-14T05:53:58.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:53:58.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:53:58.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_7 snap 2024-09-14T05:53:58.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:58.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-14T05:53:58.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:53:58.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:53:58.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_7.export 2024-09-14T05:53:58.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_7.export 2024-09-14T05:53:58.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_7.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_7.export 2024-09-14T05:53:58.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_7@snap /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_7.export 2024-09-14T05:53:58.600 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:53:58.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_7@snap /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_7.export 2024-09-14T05:53:58.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:53:58.844 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_7.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_7.export 2024-09-14T05:53:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_7.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_7.export 2024-09-14T05:53:58.929 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:53:58.929 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:53:58.929 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_8 2024-09-14T05:53:58.929 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_8 snap 2024-09-14T05:53:58.929 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:53:58.929 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:53:58.929 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-14T05:53:58.929 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:53:58.929 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_8@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T05:54:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_8 2024-09-14T05:54:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-14T05:54:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_8 started 2024-09-14T05:54:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-14T05:54:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:54:00.787 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:54:00.787 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:54:00.787 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:54:01.787 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_8 started 2024-09-14T05:54:01.787 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:01.787 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:01.787 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-14T05:54:01.787 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:54:01.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:54:01.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:54:01.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_8 2024-09-14T05:54:01.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:54:01.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:54:01.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:54:01.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:54:01.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:54:01.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:54:01.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:54:01.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:54:01.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:54:01.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:54:01.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:54:01.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_8 2024-09-14T05:54:01.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:54:01.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:54:01.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:54:01.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:54:01.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:54:01.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:54:01.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:54:01.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:54:01.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:54:01.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:54:01.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:54:01.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:54:01.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:54:01.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:54:01.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:54:01.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:54:01.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:54:01.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:54:01.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:54:01.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:54:01.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:54:01.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:54:01.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:54:01.794 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:54:01.794 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:54:01.794 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:54:01.794 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:54:01.794 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:54:01.794 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:54:01.794 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:54:01.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:54:01.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:54:01.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:54:01.814 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_8 2024-09-14T05:54:01.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2024-09-14T05:54:01.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:54:01.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:54:01.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:54:01.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_8 2024-09-14T05:54:01.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-14T05:54:01.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:54:01.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:54:01.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T05:54:01.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-14T05:54:01.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:54:01.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:54:01.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:54:01.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-14T05:54:01.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T05:54:01.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:54:01.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:54:01.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-14T05:54:01.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:54:01.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T05:54:01.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-14T05:54:01.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:54:01.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:54:01.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T05:54:01.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-14T05:54:01.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:54:01.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:54:01.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:54:01.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:54:01.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T05:54:01.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:54:01.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:54:01.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-14T05:54:01.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-14T05:54:01.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T05:54:01.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:54:01.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:54:01.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:54:01.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_8 2024-09-14T05:54:01.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2024-09-14T05:54:01.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:54:01.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:54:01.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:54:01.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_8 2024-09-14T05:54:01.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-14T05:54:01.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:54:01.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:54:01.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T05:54:01.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-14T05:54:01.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:54:01.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:54:01.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:54:01.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-14T05:54:01.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T05:54:01.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:54:01.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:54:01.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-14T05:54:01.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:54:01.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T05:54:01.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-14T05:54:01.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:54:01.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:54:01.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T05:54:01.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-14T05:54:01.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:54:01.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:54:01.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:54:01.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:54:01.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T05:54:01.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:54:01.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:54:01.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-14T05:54:01.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-14T05:54:01.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T05:54:02.011 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:54:02.011 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:54:02.011 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:54:02.011 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_8 2024-09-14T05:54:02.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:54:02.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:54:02.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:54:02.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:54:02.128 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:54:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:54:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:54:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:54:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_8 2024-09-14T05:54:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:54:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:54:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:02.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-14T05:54:02.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:54:02.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:54:02.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_8 2024-09-14T05:54:02.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:54:02.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:54:02.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:02.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-14T05:54:02.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_8.status 2024-09-14T05:54:02.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_8.status 2024-09-14T05:54:02.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:54:02.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_8.status 2024-09-14T05:54:02.132 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_8.status 2024-09-14T05:54:02.132 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_8.status 2024-09-14T05:54:02.132 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:54:02.132 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_8.status 2024-09-14T05:54:02.132 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_8 2024-09-14T05:54:02.132 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:54:02.132 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:02.132 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-14T05:54:02.132 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_8 2024-09-14T05:54:02.781 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 100 2024-09-14T05:54:02.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_8 /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_8.status 2024-09-14T05:54:02.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:54:02.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:02.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-14T05:54:02.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_8.status 2024-09-14T05:54:02.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_8 --format xml 2024-09-14T05:54:02.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:54:02.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:54:02.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=100 2024-09-14T05:54:02.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:54:02.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:54:02.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:54:03.041 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_8.status 2024-09-14T05:54:03.041 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:03.041 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:03.041 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-14T05:54:03.041 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_8.status 2024-09-14T05:54:03.041 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-09-14T05:54:03.041 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:54:03.091 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:54:03.091 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:54:03.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:54:03.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:54:03.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:54:03.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:54:03.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 100 = '' 2024-09-14T05:54:03.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:54:03.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:54:03.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_8.status 2024-09-14T05:54:03.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:03.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:03.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-14T05:54:03.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_8.status 2024-09-14T05:54:03.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-09-14T05:54:03.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:54:03.542 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:54:03.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:54:03.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:54:03.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:54:03.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:54:03.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:54:03.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 100 = '' 2024-09-14T05:54:03.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:54:03.545 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:54:04.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_8.status 2024-09-14T05:54:04.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:04.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:04.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-14T05:54:04.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_8.status 2024-09-14T05:54:04.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-09-14T05:54:04.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:54:04.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:54:04.396 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=100 2024-09-14T05:54:04.396 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 100 = 100 2024-09-14T05:54:04.396 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:54:04.396 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_8 snap 2024-09-14T05:54:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-14T05:54:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:54:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:54:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:54:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:54:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_8@snap 2024-09-14T05:54:05.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_8': 2024-09-14T05:54:05.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:54:05.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:54:05.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-14T05:54:05.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c1a580a508 2024-09-14T05:54:05.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c1a580a508 2024-09-14T05:54:05.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:54:05.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:54:05.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:54:05.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:54:05.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:45:42 2024 2024-09-14T05:54:05.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:45:42 2024 2024-09-14T05:54:05.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:45:42 2024 2024-09-14T05:54:05.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:54:05.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:54:05.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:54:05.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 21958cb0-e88b-48db-bd17-5fa7463f50e0 2024-09-14T05:54:05.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:54:05.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:54:05.450 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_8 snap 2024-09-14T05:54:05.450 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:05.450 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-14T05:54:05.450 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:54:05.450 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:54:05.450 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_8.export 2024-09-14T05:54:05.450 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_8.export 2024-09-14T05:54:05.450 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_8.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_8.export 2024-09-14T05:54:05.451 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_8@snap /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_8.export 2024-09-14T05:54:05.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:54:05.625 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_8@snap /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_8.export 2024-09-14T05:54:05.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:54:05.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_8.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_8.export 2024-09-14T05:54:05.864 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_8.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_8.export 2024-09-14T05:54:05.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:54:05.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:54:05.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_9 2024-09-14T05:54:05.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_9 snap 2024-09-14T05:54:05.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:54:05.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:05.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-14T05:54:05.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:54:05.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_9@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T05:54:07.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_9 2024-09-14T05:54:07.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:07.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:07.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-14T05:54:07.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_9 started 2024-09-14T05:54:07.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:07.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:07.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-14T05:54:07.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:54:07.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:54:07.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:54:07.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:54:08.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_9 started 2024-09-14T05:54:08.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:08.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:08.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-14T05:54:08.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:54:08.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:54:08.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:54:08.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_9 2024-09-14T05:54:08.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:54:08.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:54:08.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:54:08.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:54:08.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:54:08.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:54:08.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:54:08.808 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:54:08.808 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:54:08.808 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:54:08.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:54:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_9 2024-09-14T05:54:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:54:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:54:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:54:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:54:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:54:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:54:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:54:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:54:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:54:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:54:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:54:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:54:08.811 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:54:08.811 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:54:08.811 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:54:08.811 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:54:08.811 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:54:08.811 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:54:08.811 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:54:08.811 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:54:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:54:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:54:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:54:08.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:54:08.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:54:08.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:54:08.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:54:08.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:54:08.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:54:08.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:54:08.832 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:54:08.832 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:54:08.832 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:54:08.832 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_9 2024-09-14T05:54:08.911 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2024-09-14T05:54:08.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:54:08.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:54:08.914 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:54:08.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_9 2024-09-14T05:54:08.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-14T05:54:08.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:54:08.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:54:08.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T05:54:08.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-14T05:54:08.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:54:08.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:54:08.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:54:08.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-14T05:54:08.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T05:54:08.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:54:08.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:54:08.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-14T05:54:08.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:54:08.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T05:54:08.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-14T05:54:08.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:54:08.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:54:08.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T05:54:08.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-14T05:54:08.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:54:08.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:54:08.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:54:08.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:54:08.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T05:54:08.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:54:08.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:54:08.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-14T05:54:08.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-14T05:54:08.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T05:54:08.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:54:08.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:54:08.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:54:08.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_9 2024-09-14T05:54:09.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2024-09-14T05:54:09.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:54:09.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:54:09.011 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:54:09.011 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_9 2024-09-14T05:54:09.011 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-14T05:54:09.011 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:54:09.011 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:54:09.011 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T05:54:09.011 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-14T05:54:09.011 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:54:09.012 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:54:09.012 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:54:09.012 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-14T05:54:09.012 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T05:54:09.012 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:54:09.012 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:54:09.012 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-14T05:54:09.012 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:54:09.012 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T05:54:09.012 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-14T05:54:09.012 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:54:09.012 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:54:09.012 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T05:54:09.012 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-14T05:54:09.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:54:09.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:54:09.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:54:09.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:54:09.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T05:54:09.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:54:09.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:54:09.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-14T05:54:09.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-14T05:54:09.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T05:54:09.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:54:09.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:54:09.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:54:09.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_9 2024-09-14T05:54:09.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:54:09.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:54:09.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:54:09.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:54:09.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:54:09.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:54:09.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:54:09.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:54:09.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_9 2024-09-14T05:54:09.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:54:09.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:54:09.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:09.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-14T05:54:09.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:54:09.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:54:09.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_9 2024-09-14T05:54:09.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:54:09.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:54:09.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:09.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-14T05:54:09.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_9.status 2024-09-14T05:54:09.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_9.status 2024-09-14T05:54:09.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:54:09.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_9.status 2024-09-14T05:54:09.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_9.status 2024-09-14T05:54:09.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_9.status 2024-09-14T05:54:09.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:54:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_9.status 2024-09-14T05:54:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_9 2024-09-14T05:54:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:54:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-14T05:54:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_9 2024-09-14T05:54:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 102 2024-09-14T05:54:09.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_9 /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_9.status 2024-09-14T05:54:09.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:54:09.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:09.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-14T05:54:09.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_9.status 2024-09-14T05:54:09.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_9 --format xml 2024-09-14T05:54:09.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:54:09.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:54:09.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=102 2024-09-14T05:54:09.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:54:09.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:54:09.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:54:10.069 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_9.status 2024-09-14T05:54:10.069 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:10.069 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:10.069 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-14T05:54:10.069 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_9.status 2024-09-14T05:54:10.069 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-09-14T05:54:10.069 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:54:10.117 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:54:10.118 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:54:10.119 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:54:10.119 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:54:10.119 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:54:10.119 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:54:10.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 102 = '' 2024-09-14T05:54:10.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:54:10.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:54:10.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_9.status 2024-09-14T05:54:10.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:10.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:10.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-14T05:54:10.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_9.status 2024-09-14T05:54:10.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-09-14T05:54:10.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:54:10.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:54:10.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:54:10.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:54:10.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:54:10.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:54:10.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:54:10.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 102 = '' 2024-09-14T05:54:10.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:54:10.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:54:11.375 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_9.status 2024-09-14T05:54:11.376 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:11.376 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:11.376 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-14T05:54:11.376 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_9.status 2024-09-14T05:54:11.376 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-09-14T05:54:11.376 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:54:11.424 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:54:11.426 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=102 2024-09-14T05:54:11.426 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 102 = 102 2024-09-14T05:54:11.426 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:54:11.426 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_9 snap 2024-09-14T05:54:11.426 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:11.426 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:11.426 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-14T05:54:11.426 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:54:11.426 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:54:11.426 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:54:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:54:12.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_9@snap 2024-09-14T05:54:12.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_9': 2024-09-14T05:54:12.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:54:12.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:54:12.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-14T05:54:12.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c17b72b8 2024-09-14T05:54:12.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c17b72b8 2024-09-14T05:54:12.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:54:12.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:54:12.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:54:12.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:54:12.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:45:52 2024 2024-09-14T05:54:12.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:45:52 2024 2024-09-14T05:54:12.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:45:52 2024 2024-09-14T05:54:12.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:54:12.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:54:12.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:54:12.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 8d3c3903-631c-42c7-8d30-a5b307b883eb 2024-09-14T05:54:12.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:54:12.478 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:54:12.478 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_9 snap 2024-09-14T05:54:12.478 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:12.478 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-14T05:54:12.478 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:54:12.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:54:12.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_9.export 2024-09-14T05:54:12.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_9.export 2024-09-14T05:54:12.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_9.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_9.export 2024-09-14T05:54:12.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_9@snap /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_9.export 2024-09-14T05:54:12.657 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:54:12.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_9@snap /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_9.export 2024-09-14T05:54:12.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:54:12.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_9.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_9.export 2024-09-14T05:54:12.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_9.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_9.export 2024-09-14T05:54:12.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:54:12.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:54:12.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_10 2024-09-14T05:54:12.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_10 snap 2024-09-14T05:54:12.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:54:12.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:12.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-14T05:54:12.969 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:54:12.969 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_10@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T05:54:14.821 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_10 2024-09-14T05:54:14.821 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:14.821 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:14.821 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-14T05:54:14.821 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_10 started 2024-09-14T05:54:14.821 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:14.821 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:14.821 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-14T05:54:14.821 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:54:14.821 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:54:14.822 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:54:14.822 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:54:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_10 started 2024-09-14T05:54:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-14T05:54:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:54:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:54:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:54:15.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_10 2024-09-14T05:54:15.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:54:15.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:54:15.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:54:15.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:54:15.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:54:15.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:54:15.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:54:15.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:54:15.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:54:15.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:54:15.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:54:15.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_10 2024-09-14T05:54:15.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:54:15.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:54:15.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:54:15.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:54:15.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:54:15.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:54:15.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:54:15.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:54:15.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:54:15.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:54:15.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:54:15.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:54:15.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:54:15.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:54:15.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:54:15.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:54:15.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:54:15.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:54:15.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:54:15.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:54:15.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:54:15.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:54:15.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:54:15.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:54:15.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:54:15.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:54:15.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:54:15.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:54:15.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:54:15.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:54:15.852 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:54:15.852 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:54:15.852 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:54:15.852 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_10 2024-09-14T05:54:15.929 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-09-14T05:54:15.929 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:54:15.930 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:54:15.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:54:15.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_10 2024-09-14T05:54:15.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-14T05:54:15.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:54:15.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:54:15.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T05:54:15.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-14T05:54:15.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:54:15.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:54:15.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:54:15.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-14T05:54:15.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T05:54:15.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:54:15.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:54:15.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-14T05:54:15.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:54:15.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T05:54:15.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-14T05:54:15.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:54:15.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:54:15.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T05:54:15.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-14T05:54:15.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:54:15.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:54:15.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:54:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:54:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T05:54:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:54:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:54:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-14T05:54:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-14T05:54:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T05:54:15.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:54:15.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:54:15.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:54:15.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_10 2024-09-14T05:54:16.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-09-14T05:54:16.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:54:16.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:54:16.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:54:16.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_10 2024-09-14T05:54:16.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-14T05:54:16.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:54:16.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:54:16.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T05:54:16.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-14T05:54:16.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:54:16.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:54:16.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:54:16.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-14T05:54:16.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T05:54:16.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:54:16.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:54:16.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-14T05:54:16.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:54:16.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T05:54:16.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-14T05:54:16.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:54:16.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:54:16.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T05:54:16.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-14T05:54:16.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:54:16.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:54:16.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:54:16.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:54:16.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T05:54:16.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:54:16.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:54:16.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-14T05:54:16.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-14T05:54:16.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T05:54:16.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:54:16.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:54:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:54:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_10 2024-09-14T05:54:16.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:54:16.171 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:54:16.171 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:54:16.171 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:54:16.172 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:54:16.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:54:16.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:54:16.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:54:16.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_10 2024-09-14T05:54:16.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:54:16.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:54:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-14T05:54:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:54:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:54:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_10 2024-09-14T05:54:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:54:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:54:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-14T05:54:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_10.status 2024-09-14T05:54:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_10.status 2024-09-14T05:54:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:54:16.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_10.status 2024-09-14T05:54:16.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_10.status 2024-09-14T05:54:16.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_10.status 2024-09-14T05:54:16.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:54:16.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_10.status 2024-09-14T05:54:16.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_10 2024-09-14T05:54:16.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:54:16.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:16.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-14T05:54:16.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_10 2024-09-14T05:54:16.822 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 104 2024-09-14T05:54:16.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_10 /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_10.status 2024-09-14T05:54:16.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:54:16.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:16.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-14T05:54:16.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_10.status 2024-09-14T05:54:16.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_10 --format xml 2024-09-14T05:54:16.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:54:16.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:54:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=104 2024-09-14T05:54:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:54:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:54:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:54:17.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_10.status 2024-09-14T05:54:17.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:17.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:17.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-14T05:54:17.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_10.status 2024-09-14T05:54:17.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-09-14T05:54:17.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:54:17.132 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:54:17.132 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:54:17.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:54:17.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:54:17.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:54:17.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:54:17.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 104 = '' 2024-09-14T05:54:17.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:54:17.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:54:17.534 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_10.status 2024-09-14T05:54:17.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:17.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:17.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-14T05:54:17.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_10.status 2024-09-14T05:54:17.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-09-14T05:54:17.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:54:17.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:54:17.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:54:17.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:54:17.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:54:17.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:54:17.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:54:17.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 104 = '' 2024-09-14T05:54:17.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:54:17.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:54:18.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_10.status 2024-09-14T05:54:18.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:18.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:18.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-14T05:54:18.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_10.status 2024-09-14T05:54:18.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-09-14T05:54:18.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:54:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:54:18.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=104 2024-09-14T05:54:18.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 104 = 104 2024-09-14T05:54:18.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:54:18.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_10 snap 2024-09-14T05:54:18.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:18.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:18.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-14T05:54:18.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:54:18.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:54:18.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:54:18.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:54:19.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_10@snap 2024-09-14T05:54:19.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_10': 2024-09-14T05:54:19.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:54:19.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:54:19.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-14T05:54:19.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c14c9e92b1 2024-09-14T05:54:19.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c14c9e92b1 2024-09-14T05:54:19.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:54:19.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:54:19.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:54:19.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:54:19.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:46:05 2024 2024-09-14T05:54:19.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:46:05 2024 2024-09-14T05:54:19.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:46:05 2024 2024-09-14T05:54:19.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:54:19.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:54:19.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:54:19.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: c0427ec2-f759-4269-ac43-364d5f6678fa 2024-09-14T05:54:19.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:54:19.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:54:19.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_10 snap 2024-09-14T05:54:19.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:19.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-14T05:54:19.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:54:19.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:54:19.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_10.export 2024-09-14T05:54:19.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_10.export 2024-09-14T05:54:19.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_10.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_10.export 2024-09-14T05:54:19.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_10@snap /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_10.export 2024-09-14T05:54:19.667 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:54:19.674 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_10@snap /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_10.export 2024-09-14T05:54:19.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:54:19.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_10.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_10.export 2024-09-14T05:54:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_10.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_10.export 2024-09-14T05:54:20.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:54:20.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:54:20.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_11 2024-09-14T05:54:20.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_11 snap 2024-09-14T05:54:20.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:54:20.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:20.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_11 2024-09-14T05:54:20.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:54:20.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_11@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T05:54:21.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_11 2024-09-14T05:54:21.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:21.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:21.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_11 2024-09-14T05:54:21.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_11 started 2024-09-14T05:54:21.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:21.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:21.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_11 2024-09-14T05:54:21.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:54:21.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:54:21.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:54:21.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:54:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_11 started 2024-09-14T05:54:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_11 2024-09-14T05:54:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:54:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:54:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:54:22.856 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_11 2024-09-14T05:54:22.856 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:54:22.856 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:54:22.856 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:54:22.857 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:54:22.857 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:54:22.857 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:54:22.857 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:54:22.857 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:54:22.857 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:54:22.857 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:54:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:54:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_11 2024-09-14T05:54:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:54:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:54:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:54:22.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:54:22.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:54:22.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:54:22.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:54:22.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:54:22.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:54:22.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:54:22.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:54:22.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:54:22.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:54:22.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:54:22.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:54:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:54:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:54:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:54:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:54:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:54:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:54:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:54:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:54:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:54:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:54:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:54:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:54:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:54:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:54:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:54:22.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:54:22.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:54:22.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:54:22.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_11 2024-09-14T05:54:22.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2024-09-14T05:54:22.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:54:22.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:54:22.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:54:22.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_11 2024-09-14T05:54:22.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-14T05:54:22.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:54:22.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:54:22.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T05:54:22.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-14T05:54:22.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:54:22.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:54:22.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:54:22.964 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-14T05:54:22.964 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T05:54:22.964 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:54:22.964 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:54:22.964 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-14T05:54:22.964 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:54:22.964 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T05:54:22.964 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-14T05:54:22.964 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:54:22.964 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:54:22.964 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T05:54:22.964 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-14T05:54:22.964 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:54:22.965 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:54:22.965 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:54:22.965 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:54:22.965 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T05:54:22.965 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:54:22.965 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:54:22.965 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-14T05:54:22.965 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-14T05:54:22.965 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T05:54:22.983 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:54:22.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:54:22.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:54:22.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_11 2024-09-14T05:54:23.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2024-09-14T05:54:23.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:54:23.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:54:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:54:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_11 2024-09-14T05:54:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-14T05:54:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:54:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:54:23.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T05:54:23.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-14T05:54:23.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:54:23.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:54:23.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:54:23.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-14T05:54:23.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T05:54:23.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:54:23.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:54:23.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-14T05:54:23.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:54:23.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T05:54:23.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-14T05:54:23.062 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:54:23.062 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:54:23.062 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T05:54:23.062 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-14T05:54:23.062 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:54:23.062 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:54:23.062 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:54:23.062 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:54:23.062 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T05:54:23.062 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:54:23.062 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:54:23.062 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-14T05:54:23.062 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-14T05:54:23.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T05:54:23.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:54:23.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:54:23.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:54:23.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_11 2024-09-14T05:54:23.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:54:23.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:54:23.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:54:23.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:54:23.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:54:23.193 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:54:23.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:54:23.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:54:23.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_11 2024-09-14T05:54:23.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:54:23.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:54:23.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:23.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_11 2024-09-14T05:54:23.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:54:23.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:54:23.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_11 2024-09-14T05:54:23.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:54:23.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:54:23.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:23.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_11 2024-09-14T05:54:23.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_11.status 2024-09-14T05:54:23.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_11.status 2024-09-14T05:54:23.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:54:23.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_11.status 2024-09-14T05:54:23.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_11.status 2024-09-14T05:54:23.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_11.status 2024-09-14T05:54:23.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:54:23.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_11.status 2024-09-14T05:54:23.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_11 2024-09-14T05:54:23.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:54:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_11 2024-09-14T05:54:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_11 2024-09-14T05:54:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 106 2024-09-14T05:54:23.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_11 /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_11.status 2024-09-14T05:54:23.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:54:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_11 2024-09-14T05:54:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_11.status 2024-09-14T05:54:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_11 --format xml 2024-09-14T05:54:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:54:23.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:54:23.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=106 2024-09-14T05:54:23.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:54:23.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:54:23.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:54:24.118 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_11.status 2024-09-14T05:54:24.118 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:24.118 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:24.118 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_11 2024-09-14T05:54:24.119 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_11.status 2024-09-14T05:54:24.119 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-09-14T05:54:24.119 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:54:24.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:54:24.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:54:24.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:54:24.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:54:24.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:54:24.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:54:24.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 106 = '' 2024-09-14T05:54:24.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:54:24.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:54:24.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_11.status 2024-09-14T05:54:24.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_11 2024-09-14T05:54:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_11.status 2024-09-14T05:54:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-09-14T05:54:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:54:24.631 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:54:24.631 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:54:24.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:54:24.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:54:24.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:54:24.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:54:24.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 106 = '' 2024-09-14T05:54:24.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:54:24.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:54:25.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_11.status 2024-09-14T05:54:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_11 2024-09-14T05:54:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_11.status 2024-09-14T05:54:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-09-14T05:54:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:54:25.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:54:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=106 2024-09-14T05:54:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 106 = 106 2024-09-14T05:54:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:54:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_11 snap 2024-09-14T05:54:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_11 2024-09-14T05:54:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:54:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:54:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:54:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:54:26.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_11@snap 2024-09-14T05:54:26.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_11': 2024-09-14T05:54:26.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:54:26.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:54:26.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-14T05:54:26.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c19c83a430 2024-09-14T05:54:26.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c19c83a430 2024-09-14T05:54:26.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:54:26.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:54:26.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:54:26.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:54:26.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:46:15 2024 2024-09-14T05:54:26.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:46:15 2024 2024-09-14T05:54:26.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:46:15 2024 2024-09-14T05:54:26.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:54:26.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:54:26.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:54:26.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: b6cea04e-7839-4e62-8e00-34e757bf5550 2024-09-14T05:54:26.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:54:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:54:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_11 snap 2024-09-14T05:54:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_11 2024-09-14T05:54:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:54:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:54:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_11.export 2024-09-14T05:54:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_11.export 2024-09-14T05:54:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_11.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_11.export 2024-09-14T05:54:26.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_11@snap /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_11.export 2024-09-14T05:54:26.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:54:26.722 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_11@snap /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_11.export 2024-09-14T05:54:26.952 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:54:26.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_11.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_11.export 2024-09-14T05:54:27.014 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_11.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_11.export 2024-09-14T05:54:27.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:54:27.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:54:27.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_12 2024-09-14T05:54:27.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_12 snap 2024-09-14T05:54:27.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:54:27.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:27.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_12 2024-09-14T05:54:27.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:54:27.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_12@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T05:54:28.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_12 2024-09-14T05:54:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_12 2024-09-14T05:54:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_12 started 2024-09-14T05:54:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_12 2024-09-14T05:54:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:54:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:54:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:54:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:54:29.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_12 started 2024-09-14T05:54:29.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:29.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:29.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_12 2024-09-14T05:54:29.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:54:29.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:54:29.863 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:54:29.864 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_12 2024-09-14T05:54:29.864 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:54:29.864 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:54:29.864 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:54:29.864 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:54:29.864 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:54:29.864 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:54:29.864 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:54:29.864 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:54:29.864 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:54:29.864 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:54:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:54:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_12 2024-09-14T05:54:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:54:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:54:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:54:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:54:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:54:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:54:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:54:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:54:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:54:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:54:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:54:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:54:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:54:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:54:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:54:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:54:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:54:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:54:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:54:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:54:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:54:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:54:29.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:54:29.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:54:29.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:54:29.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:54:29.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:54:29.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:54:29.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:54:29.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:54:29.890 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:54:29.890 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:54:29.890 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:54:29.890 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_12 2024-09-14T05:54:29.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-09-14T05:54:29.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:54:29.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:54:29.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:54:29.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_12 2024-09-14T05:54:29.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-14T05:54:29.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:54:29.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:54:29.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T05:54:29.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-14T05:54:29.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:54:29.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:54:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:54:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-14T05:54:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T05:54:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:54:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:54:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-14T05:54:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:54:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T05:54:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-14T05:54:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:54:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:54:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T05:54:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-14T05:54:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:54:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:54:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:54:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:54:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T05:54:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:54:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:54:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-14T05:54:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-14T05:54:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T05:54:29.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:54:29.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:54:29.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:54:29.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_12 2024-09-14T05:54:30.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-09-14T05:54:30.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:54:30.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:54:30.069 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:54:30.069 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_12 2024-09-14T05:54:30.069 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-14T05:54:30.069 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:54:30.069 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:54:30.069 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T05:54:30.069 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-14T05:54:30.069 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:54:30.069 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:54:30.069 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:54:30.070 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-14T05:54:30.070 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T05:54:30.070 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:54:30.070 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:54:30.070 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-14T05:54:30.070 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:54:30.070 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T05:54:30.070 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-14T05:54:30.070 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:54:30.070 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:54:30.070 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T05:54:30.070 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-14T05:54:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:54:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:54:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:54:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:54:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T05:54:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:54:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:54:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-14T05:54:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-14T05:54:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T05:54:30.088 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:54:30.088 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:54:30.088 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:54:30.088 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_12 2024-09-14T05:54:30.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:54:30.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:54:30.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:54:30.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:54:30.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:54:30.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:54:30.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:54:30.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:54:30.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_12 2024-09-14T05:54:30.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:54:30.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:54:30.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:30.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_12 2024-09-14T05:54:30.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:54:30.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:54:30.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_12 2024-09-14T05:54:30.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:54:30.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:54:30.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:30.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_12 2024-09-14T05:54:30.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_12.status 2024-09-14T05:54:30.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_12.status 2024-09-14T05:54:30.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:54:30.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_12.status 2024-09-14T05:54:30.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_12.status 2024-09-14T05:54:30.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_12.status 2024-09-14T05:54:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:54:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_12.status 2024-09-14T05:54:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_12 2024-09-14T05:54:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:54:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_12 2024-09-14T05:54:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_12 2024-09-14T05:54:30.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 108 2024-09-14T05:54:30.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_12 /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_12.status 2024-09-14T05:54:30.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:54:30.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:30.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_12 2024-09-14T05:54:30.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_12.status 2024-09-14T05:54:30.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_12 --format xml 2024-09-14T05:54:30.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:54:30.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:54:30.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=108 2024-09-14T05:54:30.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:54:30.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:54:30.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:54:31.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_12.status 2024-09-14T05:54:31.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:31.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:31.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_12 2024-09-14T05:54:31.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_12.status 2024-09-14T05:54:31.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-09-14T05:54:31.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:54:31.172 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:54:31.172 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:54:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:54:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:54:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:54:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:54:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 108 = '' 2024-09-14T05:54:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:54:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:54:31.505 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.960769359489311) > weight (0.25) 2024-09-14T05:54:31.505 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.6872507456954638) > weight (0.25) 2024-09-14T05:54:31.505 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.48825563962189056) > weight (0.25) 2024-09-14T05:54:31.505 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.49905042332807614) > weight (0.25) 2024-09-14T05:54:31.506 INFO:tasks.rbd_mirror_thrash:waiting for 63 secs before thrashing 2024-09-14T05:54:31.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_12.status 2024-09-14T05:54:31.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:31.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:31.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_12 2024-09-14T05:54:31.577 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_12.status 2024-09-14T05:54:31.577 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-09-14T05:54:31.577 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:54:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:54:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:54:31.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:54:31.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:54:31.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:54:31.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:54:31.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 108 = '' 2024-09-14T05:54:31.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:54:31.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:54:32.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_12.status 2024-09-14T05:54:32.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:32.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:32.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_12 2024-09-14T05:54:32.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_12.status 2024-09-14T05:54:32.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-09-14T05:54:32.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:54:32.477 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:54:32.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=108 2024-09-14T05:54:32.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 108 = 108 2024-09-14T05:54:32.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:54:32.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_12 snap 2024-09-14T05:54:32.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:32.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:32.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_12 2024-09-14T05:54:32.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:54:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:54:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:54:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:54:33.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_12@snap 2024-09-14T05:54:33.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_12': 2024-09-14T05:54:33.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:54:33.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:54:33.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-14T05:54:33.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c1bb599cb5 2024-09-14T05:54:33.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c1bb599cb5 2024-09-14T05:54:33.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:54:33.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:54:33.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:54:33.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:54:33.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:46:28 2024 2024-09-14T05:54:33.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:46:28 2024 2024-09-14T05:54:33.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:46:28 2024 2024-09-14T05:54:33.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:54:33.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:54:33.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:54:33.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 7b610a90-6bb9-4ffb-8270-bbf747673d2a 2024-09-14T05:54:33.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:54:33.534 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:54:33.534 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_12 snap 2024-09-14T05:54:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_12 2024-09-14T05:54:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:54:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:54:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_12.export 2024-09-14T05:54:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_12.export 2024-09-14T05:54:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_12.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_12.export 2024-09-14T05:54:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_12@snap /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_12.export 2024-09-14T05:54:33.700 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:54:33.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_12@snap /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_12.export 2024-09-14T05:54:33.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:54:33.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_12.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_12.export 2024-09-14T05:54:33.999 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_12.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_12.export 2024-09-14T05:54:34.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:54:34.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:54:34.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_13 2024-09-14T05:54:34.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_13 snap 2024-09-14T05:54:34.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:54:34.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:34.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_13 2024-09-14T05:54:34.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:54:34.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_13@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T05:54:35.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_13 2024-09-14T05:54:35.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:35.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:35.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_13 2024-09-14T05:54:35.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_13 started 2024-09-14T05:54:35.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:35.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:35.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_13 2024-09-14T05:54:35.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:54:35.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:54:35.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:54:35.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:54:36.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_13 started 2024-09-14T05:54:36.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:36.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:36.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_13 2024-09-14T05:54:36.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:54:36.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:54:36.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:54:36.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_13 2024-09-14T05:54:36.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:54:36.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:54:36.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:54:36.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:54:36.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:54:36.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:54:36.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:54:36.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:54:36.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:54:36.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:54:36.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:54:36.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_13 2024-09-14T05:54:36.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:54:36.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:54:36.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:54:36.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:54:36.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:54:36.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:54:36.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:54:36.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:54:36.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:54:36.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:54:36.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:54:36.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:54:36.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:54:36.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:54:36.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:54:36.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:54:36.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:54:36.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:54:36.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:54:36.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:54:36.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:54:36.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:54:36.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:54:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:54:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:54:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:54:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:54:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:54:36.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:54:36.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:54:36.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:54:36.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:54:36.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:54:36.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_13 2024-09-14T05:54:36.983 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2024-09-14T05:54:36.983 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:54:36.983 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:54:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:54:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_13 2024-09-14T05:54:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-14T05:54:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:54:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:54:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T05:54:36.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-14T05:54:36.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:54:36.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:54:36.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:54:36.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-14T05:54:36.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T05:54:36.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:54:36.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:54:36.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-14T05:54:36.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:54:36.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T05:54:36.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-14T05:54:36.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:54:36.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:54:36.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T05:54:36.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-14T05:54:36.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:54:36.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:54:36.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:54:36.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:54:36.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T05:54:36.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:54:36.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:54:36.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-14T05:54:36.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-14T05:54:36.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T05:54:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:54:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:54:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:54:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_13 2024-09-14T05:54:37.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2024-09-14T05:54:37.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:54:37.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:54:37.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:54:37.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_13 2024-09-14T05:54:37.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-14T05:54:37.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:54:37.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:54:37.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T05:54:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-14T05:54:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:54:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:54:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:54:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-14T05:54:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T05:54:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:54:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:54:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-14T05:54:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:54:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T05:54:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-14T05:54:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:54:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:54:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T05:54:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-14T05:54:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:54:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:54:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:54:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:54:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T05:54:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:54:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:54:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-14T05:54:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-14T05:54:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T05:54:37.101 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:54:37.101 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:54:37.101 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:54:37.101 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_13 2024-09-14T05:54:37.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:54:37.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:54:37.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:54:37.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:54:37.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:54:37.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:54:37.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:54:37.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:54:37.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_13 2024-09-14T05:54:37.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:54:37.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:54:37.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:37.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_13 2024-09-14T05:54:37.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:54:37.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:54:37.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_13 2024-09-14T05:54:37.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:54:37.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:54:37.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:37.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_13 2024-09-14T05:54:37.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_13.status 2024-09-14T05:54:37.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_13.status 2024-09-14T05:54:37.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:54:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_13.status 2024-09-14T05:54:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_13.status 2024-09-14T05:54:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_13.status 2024-09-14T05:54:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:54:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_13.status 2024-09-14T05:54:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_13 2024-09-14T05:54:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:54:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_13 2024-09-14T05:54:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_13 2024-09-14T05:54:37.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 110 2024-09-14T05:54:37.888 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_13 /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_13.status 2024-09-14T05:54:37.889 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:54:37.889 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:37.889 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_13 2024-09-14T05:54:37.889 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_13.status 2024-09-14T05:54:37.889 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_13 --format xml 2024-09-14T05:54:37.889 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:54:37.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:54:37.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=110 2024-09-14T05:54:37.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:54:37.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:54:37.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:54:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_13.status 2024-09-14T05:54:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_13 2024-09-14T05:54:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_13.status 2024-09-14T05:54:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-09-14T05:54:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:54:38.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:54:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:54:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:54:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:54:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:54:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:54:38.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 110 = '' 2024-09-14T05:54:38.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:54:38.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:54:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_13.status 2024-09-14T05:54:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_13 2024-09-14T05:54:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_13.status 2024-09-14T05:54:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-09-14T05:54:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:54:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:54:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:54:38.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:54:38.645 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:54:38.645 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:54:38.645 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:54:38.645 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 110 = '' 2024-09-14T05:54:38.645 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:54:38.645 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:54:39.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_13.status 2024-09-14T05:54:39.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:39.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:39.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_13 2024-09-14T05:54:39.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_13.status 2024-09-14T05:54:39.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-09-14T05:54:39.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:54:39.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:54:39.498 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=110 2024-09-14T05:54:39.498 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 110 = 110 2024-09-14T05:54:39.498 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:54:39.498 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_13 snap 2024-09-14T05:54:39.498 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:39.498 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:39.499 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_13 2024-09-14T05:54:39.499 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:54:39.499 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:54:39.499 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:54:39.499 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:54:40.499 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_13@snap 2024-09-14T05:54:40.546 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_13': 2024-09-14T05:54:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:54:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:54:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-14T05:54:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c1d24a8ddc 2024-09-14T05:54:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c1d24a8ddc 2024-09-14T05:54:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:54:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:54:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:54:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:54:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:46:37 2024 2024-09-14T05:54:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:46:37 2024 2024-09-14T05:54:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:46:37 2024 2024-09-14T05:54:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:54:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:54:40.548 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:54:40.548 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 03d4b3e9-bc59-4da0-9ad0-74a331dde7bb 2024-09-14T05:54:40.548 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:54:40.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:54:40.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_13 snap 2024-09-14T05:54:40.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:40.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_13 2024-09-14T05:54:40.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:54:40.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:54:40.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_13.export 2024-09-14T05:54:40.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_13.export 2024-09-14T05:54:40.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_13.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_13.export 2024-09-14T05:54:40.552 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_13@snap /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_13.export 2024-09-14T05:54:40.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:54:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_13@snap /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_13.export 2024-09-14T05:54:40.965 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:54:40.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_13.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_13.export 2024-09-14T05:54:41.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_13.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_13.export 2024-09-14T05:54:41.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:54:41.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:54:41.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_14 2024-09-14T05:54:41.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_14 snap 2024-09-14T05:54:41.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:54:41.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:41.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_14 2024-09-14T05:54:41.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:54:41.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_14@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T05:54:42.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_14 2024-09-14T05:54:42.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:42.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:42.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_14 2024-09-14T05:54:42.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_14 started 2024-09-14T05:54:42.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:42.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_14 2024-09-14T05:54:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:54:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:54:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:54:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:54:43.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_14 started 2024-09-14T05:54:43.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:43.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:43.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_14 2024-09-14T05:54:43.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:54:43.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:54:43.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:54:43.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_14 2024-09-14T05:54:43.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:54:43.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:54:43.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:54:43.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:54:43.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:54:43.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:54:43.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:54:43.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:54:43.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:54:43.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:54:43.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:54:43.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_14 2024-09-14T05:54:43.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:54:43.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:54:43.899 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:54:43.899 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:54:43.899 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:54:43.899 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:54:43.899 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:54:43.899 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:54:43.899 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:54:43.899 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:54:43.899 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:54:43.899 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:54:43.899 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:54:43.899 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:54:43.900 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:54:43.900 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:54:43.900 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:54:43.900 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:54:43.900 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:54:43.900 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:54:43.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:54:43.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:54:43.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:54:43.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:54:43.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:54:43.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:54:43.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:54:43.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:54:43.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:54:43.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:54:43.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:54:43.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:54:43.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:54:43.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_14 2024-09-14T05:54:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-09-14T05:54:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:54:44.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:54:44.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:54:44.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_14 2024-09-14T05:54:44.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-14T05:54:44.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:54:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:54:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T05:54:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-14T05:54:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:54:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:54:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:54:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-14T05:54:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T05:54:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:54:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:54:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-14T05:54:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:54:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T05:54:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-14T05:54:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:54:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:54:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T05:54:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-14T05:54:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:54:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:54:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:54:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:54:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T05:54:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:54:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:54:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-14T05:54:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-14T05:54:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T05:54:44.021 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:54:44.021 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:54:44.021 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:54:44.021 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_14 2024-09-14T05:54:44.094 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-09-14T05:54:44.094 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:54:44.094 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:54:44.097 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:54:44.097 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_14 2024-09-14T05:54:44.097 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-14T05:54:44.097 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:54:44.097 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:54:44.097 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T05:54:44.097 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-14T05:54:44.097 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:54:44.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:54:44.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:54:44.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-14T05:54:44.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T05:54:44.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:54:44.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:54:44.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-14T05:54:44.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:54:44.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T05:54:44.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-14T05:54:44.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:54:44.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:54:44.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T05:54:44.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-14T05:54:44.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:54:44.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:54:44.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:54:44.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:54:44.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T05:54:44.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:54:44.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:54:44.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-14T05:54:44.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-14T05:54:44.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T05:54:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:54:44.117 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:54:44.117 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:54:44.117 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_14 2024-09-14T05:54:44.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:54:44.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:54:44.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:54:44.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:54:44.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:54:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:54:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:54:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:54:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_14 2024-09-14T05:54:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:54:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:54:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_14 2024-09-14T05:54:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:54:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:54:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_14 2024-09-14T05:54:44.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:54:44.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:54:44.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:44.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_14 2024-09-14T05:54:44.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_14.status 2024-09-14T05:54:44.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_14.status 2024-09-14T05:54:44.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:54:44.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_14.status 2024-09-14T05:54:44.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_14.status 2024-09-14T05:54:44.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_14.status 2024-09-14T05:54:44.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:54:44.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_14.status 2024-09-14T05:54:44.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_14 2024-09-14T05:54:44.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:54:44.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:44.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_14 2024-09-14T05:54:44.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_14 2024-09-14T05:54:44.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 112 2024-09-14T05:54:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_14 /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_14.status 2024-09-14T05:54:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:54:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_14 2024-09-14T05:54:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_14.status 2024-09-14T05:54:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_14 --format xml 2024-09-14T05:54:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:54:44.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:54:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=112 2024-09-14T05:54:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:54:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:54:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:54:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_14.status 2024-09-14T05:54:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_14 2024-09-14T05:54:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_14.status 2024-09-14T05:54:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-09-14T05:54:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:54:45.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:54:45.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:54:45.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:54:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:54:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:54:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:54:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 112 = '' 2024-09-14T05:54:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:54:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:54:45.606 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_14.status 2024-09-14T05:54:45.606 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:45.606 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:45.606 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_14 2024-09-14T05:54:45.606 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_14.status 2024-09-14T05:54:45.606 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-09-14T05:54:45.606 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:54:45.657 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:54:45.657 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:54:45.658 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:54:45.658 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:54:45.658 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:54:45.659 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:54:45.659 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 112 = '' 2024-09-14T05:54:45.659 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:54:45.659 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:54:46.460 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_14.status 2024-09-14T05:54:46.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:46.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:46.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_14 2024-09-14T05:54:46.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_14.status 2024-09-14T05:54:46.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-09-14T05:54:46.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:54:46.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:54:46.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=112 2024-09-14T05:54:46.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 112 = 112 2024-09-14T05:54:46.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:54:46.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_14 snap 2024-09-14T05:54:46.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:46.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:46.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_14 2024-09-14T05:54:46.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:54:46.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:54:46.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:54:46.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:54:47.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_14@snap 2024-09-14T05:54:47.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_14': 2024-09-14T05:54:47.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:54:47.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:54:47.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-14T05:54:47.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c14559dce0 2024-09-14T05:54:47.561 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c14559dce0 2024-09-14T05:54:47.561 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:54:47.561 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:54:47.561 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:54:47.561 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:54:47.561 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:46:45 2024 2024-09-14T05:54:47.561 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:46:45 2024 2024-09-14T05:54:47.561 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:46:45 2024 2024-09-14T05:54:47.561 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:54:47.561 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:54:47.562 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:54:47.562 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 72651fba-3b42-454d-a354-6f5b7be1dd8d 2024-09-14T05:54:47.562 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:54:47.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:54:47.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_14 snap 2024-09-14T05:54:47.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:47.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_14 2024-09-14T05:54:47.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:54:47.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:54:47.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_14.export 2024-09-14T05:54:47.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_14.export 2024-09-14T05:54:47.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_14.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_14.export 2024-09-14T05:54:47.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_14@snap /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_14.export 2024-09-14T05:54:47.734 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:54:47.748 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_14@snap /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_14.export 2024-09-14T05:54:47.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:54:47.966 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_14.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_14.export 2024-09-14T05:54:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_14.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_14.export 2024-09-14T05:54:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:54:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:54:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_15 2024-09-14T05:54:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_15 snap 2024-09-14T05:54:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:54:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_15 2024-09-14T05:54:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:54:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_15@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T05:54:49.914 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_15 2024-09-14T05:54:49.914 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:49.914 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_15 2024-09-14T05:54:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_15 started 2024-09-14T05:54:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_15 2024-09-14T05:54:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:54:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:54:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:54:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:54:50.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_15 started 2024-09-14T05:54:50.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:50.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:50.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_15 2024-09-14T05:54:50.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:54:50.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:54:50.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:54:50.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_15 2024-09-14T05:54:50.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:54:50.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:54:50.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:54:50.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:54:50.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:54:50.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:54:50.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:54:50.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:54:50.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:54:50.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:54:50.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:54:50.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_15 2024-09-14T05:54:50.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:54:50.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:54:50.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:54:50.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:54:50.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:54:50.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:54:50.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:54:50.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:54:50.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:54:50.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:54:50.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:54:50.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:54:50.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:54:50.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:54:50.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:54:50.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:54:50.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:54:50.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:54:50.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:54:50.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:54:50.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:54:50.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:54:50.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:54:50.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:54:50.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:54:50.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:54:50.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:54:50.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:54:50.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:54:50.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:54:50.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:54:50.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:54:50.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:54:50.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_15 2024-09-14T05:54:51.022 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2024-09-14T05:54:51.022 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:54:51.022 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:54:51.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:54:51.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_15 2024-09-14T05:54:51.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-14T05:54:51.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:54:51.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:54:51.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T05:54:51.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-14T05:54:51.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:54:51.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:54:51.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:54:51.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-14T05:54:51.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T05:54:51.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:54:51.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:54:51.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-14T05:54:51.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:54:51.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T05:54:51.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-14T05:54:51.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:54:51.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:54:51.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T05:54:51.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-14T05:54:51.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:54:51.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:54:51.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:54:51.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:54:51.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T05:54:51.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:54:51.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:54:51.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-14T05:54:51.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-14T05:54:51.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T05:54:51.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:54:51.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:54:51.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:54:51.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_15 2024-09-14T05:54:51.117 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2024-09-14T05:54:51.117 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:54:51.117 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:54:51.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:54:51.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_15 2024-09-14T05:54:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-14T05:54:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:54:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:54:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T05:54:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-14T05:54:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:54:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:54:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:54:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-14T05:54:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T05:54:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:54:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:54:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-14T05:54:51.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:54:51.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T05:54:51.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-14T05:54:51.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:54:51.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:54:51.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T05:54:51.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-14T05:54:51.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:54:51.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:54:51.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:54:51.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:54:51.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T05:54:51.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:54:51.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:54:51.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-14T05:54:51.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-14T05:54:51.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T05:54:51.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:54:51.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:54:51.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:54:51.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_15 2024-09-14T05:54:51.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:54:51.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:54:51.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:54:51.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:54:51.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:54:51.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:54:51.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:54:51.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:54:51.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_15 2024-09-14T05:54:51.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:54:51.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:54:51.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:51.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_15 2024-09-14T05:54:51.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:54:51.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:54:51.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_15 2024-09-14T05:54:51.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:54:51.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:54:51.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:51.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_15 2024-09-14T05:54:51.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_15.status 2024-09-14T05:54:51.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_15.status 2024-09-14T05:54:51.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:54:51.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_15.status 2024-09-14T05:54:51.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_15.status 2024-09-14T05:54:51.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_15.status 2024-09-14T05:54:51.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:54:51.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_15.status 2024-09-14T05:54:51.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_15 2024-09-14T05:54:51.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:54:51.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:51.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_15 2024-09-14T05:54:51.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_15 2024-09-14T05:54:51.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 114 2024-09-14T05:54:51.926 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_15 /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_15.status 2024-09-14T05:54:51.926 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:54:51.926 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:51.926 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_15 2024-09-14T05:54:51.926 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_15.status 2024-09-14T05:54:51.926 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_15 --format xml 2024-09-14T05:54:51.926 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:54:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:54:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=114 2024-09-14T05:54:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:54:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:54:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:54:52.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_15.status 2024-09-14T05:54:52.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:52.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:52.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_15 2024-09-14T05:54:52.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_15.status 2024-09-14T05:54:52.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-09-14T05:54:52.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:54:52.226 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:54:52.226 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:54:52.228 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:54:52.228 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:54:52.228 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:54:52.228 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:54:52.228 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 114 = '' 2024-09-14T05:54:52.228 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:54:52.228 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:54:52.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_15.status 2024-09-14T05:54:52.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:52.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:52.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_15 2024-09-14T05:54:52.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_15.status 2024-09-14T05:54:52.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-09-14T05:54:52.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:54:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:54:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:54:52.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:54:52.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:54:52.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:54:52.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:54:52.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 114 = '' 2024-09-14T05:54:52.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:54:52.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:54:53.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_15.status 2024-09-14T05:54:53.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:53.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:53.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_15 2024-09-14T05:54:53.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_15.status 2024-09-14T05:54:53.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-09-14T05:54:53.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:54:53.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:54:53.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=114 2024-09-14T05:54:53.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 114 = 114 2024-09-14T05:54:53.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:54:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_15 snap 2024-09-14T05:54:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_15 2024-09-14T05:54:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:54:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:54:53.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:54:53.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:54:54.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_15@snap 2024-09-14T05:54:54.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_15': 2024-09-14T05:54:54.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:54:54.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:54:54.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-14T05:54:54.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c166037626 2024-09-14T05:54:54.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c166037626 2024-09-14T05:54:54.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:54:54.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:54:54.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:54:54.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:54:54.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:46:51 2024 2024-09-14T05:54:54.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:46:51 2024 2024-09-14T05:54:54.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:46:51 2024 2024-09-14T05:54:54.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:54:54.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:54:54.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:54:54.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 8e5e2fa0-ccc2-4850-a80c-d8985387aa83 2024-09-14T05:54:54.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:54:54.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:54:54.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_15 snap 2024-09-14T05:54:54.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:54.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_15 2024-09-14T05:54:54.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:54:54.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:54:54.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_15.export 2024-09-14T05:54:54.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_15.export 2024-09-14T05:54:54.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_15.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_15.export 2024-09-14T05:54:54.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_15@snap /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_15.export 2024-09-14T05:54:54.751 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:54:54.758 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_15@snap /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_15.export 2024-09-14T05:54:54.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:54:54.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_15.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_15.export 2024-09-14T05:54:55.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_15.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_15.export 2024-09-14T05:54:55.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:54:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:54:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_16 2024-09-14T05:54:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_16 snap 2024-09-14T05:54:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:54:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_16 2024-09-14T05:54:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:54:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_16@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T05:54:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_16 2024-09-14T05:54:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_16 2024-09-14T05:54:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_16 started 2024-09-14T05:54:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:56.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_16 2024-09-14T05:54:56.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:54:56.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:54:56.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:54:56.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:54:57.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_16 started 2024-09-14T05:54:57.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:57.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:57.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_16 2024-09-14T05:54:57.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:54:57.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:54:57.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:54:57.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_16 2024-09-14T05:54:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:54:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:54:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:54:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:54:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:54:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:54:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:54:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:54:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:54:57.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:54:57.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:54:57.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_16 2024-09-14T05:54:57.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:54:57.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:54:57.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:54:57.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:54:57.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:54:57.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:54:57.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:54:57.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:54:57.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:54:57.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:54:57.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:54:57.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:54:57.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:54:57.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:54:57.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:54:57.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:54:57.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:54:57.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:54:57.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:54:57.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:54:57.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:54:57.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:54:57.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:54:57.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:54:57.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:54:57.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:54:57.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:54:57.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:54:57.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:54:57.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:54:57.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:54:57.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:54:57.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:54:57.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_16 2024-09-14T05:54:58.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-09-14T05:54:58.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:54:58.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:54:58.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:54:58.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_16 2024-09-14T05:54:58.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-14T05:54:58.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:54:58.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:54:58.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T05:54:58.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-14T05:54:58.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:54:58.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:54:58.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:54:58.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-14T05:54:58.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T05:54:58.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:54:58.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:54:58.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-14T05:54:58.046 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:54:58.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T05:54:58.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-14T05:54:58.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:54:58.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:54:58.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T05:54:58.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-14T05:54:58.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:54:58.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:54:58.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:54:58.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:54:58.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T05:54:58.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:54:58.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:54:58.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-14T05:54:58.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-14T05:54:58.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T05:54:58.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:54:58.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:54:58.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:54:58.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_16 2024-09-14T05:54:58.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-09-14T05:54:58.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:54:58.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:54:58.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:54:58.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_16 2024-09-14T05:54:58.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-14T05:54:58.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:54:58.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:54:58.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T05:54:58.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-14T05:54:58.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:54:58.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:54:58.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:54:58.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-14T05:54:58.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T05:54:58.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:54:58.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:54:58.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-14T05:54:58.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:54:58.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T05:54:58.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-14T05:54:58.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:54:58.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:54:58.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T05:54:58.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-14T05:54:58.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:54:58.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:54:58.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:54:58.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:54:58.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T05:54:58.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:54:58.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:54:58.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-14T05:54:58.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-14T05:54:58.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T05:54:58.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:54:58.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:54:58.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:54:58.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_16 2024-09-14T05:54:58.272 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:54:58.273 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:54:58.273 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:54:58.273 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:54:58.273 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:54:58.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:54:58.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:54:58.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:54:58.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_16 2024-09-14T05:54:58.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:54:58.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:54:58.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:58.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_16 2024-09-14T05:54:58.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:54:58.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:54:58.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_16 2024-09-14T05:54:58.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:54:58.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:54:58.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:58.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_16 2024-09-14T05:54:58.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_16.status 2024-09-14T05:54:58.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_16.status 2024-09-14T05:54:58.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:54:58.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_16.status 2024-09-14T05:54:58.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_16.status 2024-09-14T05:54:58.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_16.status 2024-09-14T05:54:58.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:54:58.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_16.status 2024-09-14T05:54:58.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_16 2024-09-14T05:54:58.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:54:58.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:58.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_16 2024-09-14T05:54:58.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_16 2024-09-14T05:54:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 116 2024-09-14T05:54:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_16 /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_16.status 2024-09-14T05:54:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:54:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_16 2024-09-14T05:54:58.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_16.status 2024-09-14T05:54:58.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_16 --format xml 2024-09-14T05:54:58.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:54:59.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:54:59.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=116 2024-09-14T05:54:59.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:54:59.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:54:59.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:54:59.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_16.status 2024-09-14T05:54:59.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:59.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:59.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_16 2024-09-14T05:54:59.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_16.status 2024-09-14T05:54:59.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-09-14T05:54:59.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:54:59.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:54:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=50 2024-09-14T05:54:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 116 = 50 2024-09-14T05:54:59.270 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:54:59.270 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:54:59.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_16.status 2024-09-14T05:54:59.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:54:59.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:54:59.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_16 2024-09-14T05:54:59.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_16.status 2024-09-14T05:54:59.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-09-14T05:54:59.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:54:59.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:54:59.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=50 2024-09-14T05:54:59.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 116 = 50 2024-09-14T05:54:59.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:54:59.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:55:00.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_16.status 2024-09-14T05:55:00.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:55:00.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:00.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_16 2024-09-14T05:55:00.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_16.status 2024-09-14T05:55:00.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-09-14T05:55:00.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:55:00.588 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:55:00.588 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:55:00.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:55:00.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:55:00.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:55:00.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:55:00.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 116 = '' 2024-09-14T05:55:00.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:55:00.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-14T05:55:02.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_16.status 2024-09-14T05:55:02.193 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:55:02.193 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:02.193 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_16 2024-09-14T05:55:02.193 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_16.status 2024-09-14T05:55:02.193 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-09-14T05:55:02.193 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:55:02.247 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:55:02.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=116 2024-09-14T05:55:02.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 116 = 116 2024-09-14T05:55:02.250 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:55:02.250 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_16 snap 2024-09-14T05:55:02.250 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:55:02.250 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:02.250 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_16 2024-09-14T05:55:02.250 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:55:02.250 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:55:02.250 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:55:02.250 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:55:03.250 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_16@snap 2024-09-14T05:55:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_16': 2024-09-14T05:55:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:55:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:55:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-14T05:55:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c1429aa636 2024-09-14T05:55:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c1429aa636 2024-09-14T05:55:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:55:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:55:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:55:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:55:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:47:01 2024 2024-09-14T05:55:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:47:01 2024 2024-09-14T05:55:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:47:01 2024 2024-09-14T05:55:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:55:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:55:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:55:03.301 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 1fcc0d0b-34ef-46ef-b563-beb55715aefe 2024-09-14T05:55:03.301 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:55:03.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:55:03.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_16 snap 2024-09-14T05:55:03.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:03.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_16 2024-09-14T05:55:03.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:55:03.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:55:03.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_16.export 2024-09-14T05:55:03.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_16.export 2024-09-14T05:55:03.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_16.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_16.export 2024-09-14T05:55:03.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_16@snap /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_16.export 2024-09-14T05:55:03.472 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:55:03.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_16@snap /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_16.export 2024-09-14T05:55:03.710 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:55:03.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_16.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_16.export 2024-09-14T05:55:03.771 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_16.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_16.export 2024-09-14T05:55:03.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:55:03.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:55:03.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_17 2024-09-14T05:55:03.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_17 snap 2024-09-14T05:55:03.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:55:03.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:03.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_17 2024-09-14T05:55:03.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:55:03.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_17@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T05:55:04.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_17 2024-09-14T05:55:04.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:55:04.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:04.969 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_17 2024-09-14T05:55:04.969 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_17 started 2024-09-14T05:55:04.969 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:55:04.969 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:04.969 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_17 2024-09-14T05:55:04.969 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:55:04.969 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:55:04.969 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:55:04.969 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:55:05.969 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_17 started 2024-09-14T05:55:05.969 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:55:05.969 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:05.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_17 2024-09-14T05:55:05.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:55:05.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:55:05.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:55:05.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_17 2024-09-14T05:55:05.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:55:05.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:55:05.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:55:05.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:55:05.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:55:05.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:55:05.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:55:05.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:55:05.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:55:05.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:55:05.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:55:05.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_17 2024-09-14T05:55:05.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:55:05.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:55:05.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:55:05.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:55:05.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:55:05.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:55:05.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:55:05.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:55:05.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:55:05.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:55:05.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:55:05.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:55:05.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:55:05.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:55:05.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:55:05.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:55:05.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:55:05.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:55:05.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:55:05.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:55:05.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:55:05.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:55:05.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:55:05.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:55:05.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:55:05.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:55:05.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:55:05.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:55:05.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:55:05.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:55:05.995 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:55:05.995 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:55:05.996 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:55:05.996 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_17 2024-09-14T05:55:06.073 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2024-09-14T05:55:06.073 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:55:06.073 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:55:06.076 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:55:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_17 2024-09-14T05:55:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-14T05:55:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:55:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:55:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T05:55:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-14T05:55:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:55:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:55:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:55:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-14T05:55:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T05:55:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:55:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:55:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-14T05:55:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:55:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T05:55:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-14T05:55:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:55:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:55:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T05:55:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-14T05:55:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:55:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:55:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:55:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:55:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T05:55:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:55:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:55:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-14T05:55:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-14T05:55:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T05:55:06.096 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:55:06.096 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:55:06.096 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:55:06.096 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_17 2024-09-14T05:55:06.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2024-09-14T05:55:06.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:55:06.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:55:06.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:55:06.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_17 2024-09-14T05:55:06.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-14T05:55:06.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:55:06.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:55:06.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T05:55:06.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-14T05:55:06.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:55:06.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:55:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:55:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-14T05:55:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T05:55:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:55:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:55:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-14T05:55:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:55:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T05:55:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-14T05:55:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:55:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:55:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T05:55:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-14T05:55:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:55:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:55:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:55:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:55:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T05:55:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:55:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:55:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-14T05:55:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-14T05:55:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T05:55:06.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:55:06.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:55:06.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:55:06.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_17 2024-09-14T05:55:06.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:55:06.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:55:06.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:55:06.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:55:06.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:55:06.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:55:06.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:55:06.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:55:06.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_17 2024-09-14T05:55:06.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:55:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:55:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_17 2024-09-14T05:55:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:55:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:55:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_17 2024-09-14T05:55:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:55:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:55:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_17 2024-09-14T05:55:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_17.status 2024-09-14T05:55:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_17.status 2024-09-14T05:55:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:55:06.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_17.status 2024-09-14T05:55:06.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_17.status 2024-09-14T05:55:06.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_17.status 2024-09-14T05:55:06.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:55:06.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_17.status 2024-09-14T05:55:06.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_17 2024-09-14T05:55:06.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:55:06.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:06.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_17 2024-09-14T05:55:06.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_17 2024-09-14T05:55:06.978 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 118 2024-09-14T05:55:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_17 /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_17.status 2024-09-14T05:55:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:55:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_17 2024-09-14T05:55:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_17.status 2024-09-14T05:55:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_17 --format xml 2024-09-14T05:55:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:55:07.037 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:55:07.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=118 2024-09-14T05:55:07.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:55:07.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:55:07.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:55:07.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_17.status 2024-09-14T05:55:07.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:55:07.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:07.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_17 2024-09-14T05:55:07.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_17.status 2024-09-14T05:55:07.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-09-14T05:55:07.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:55:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:55:07.293 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=51 2024-09-14T05:55:07.293 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 118 = 51 2024-09-14T05:55:07.293 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:55:07.293 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:55:07.694 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_17.status 2024-09-14T05:55:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:55:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_17 2024-09-14T05:55:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_17.status 2024-09-14T05:55:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-09-14T05:55:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:55:07.744 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:55:07.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=51 2024-09-14T05:55:07.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 118 = 51 2024-09-14T05:55:07.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:55:07.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:55:08.494 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7610924103804916) > weight (0.25) 2024-09-14T05:55:08.494 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.8408495636595779) > weight (0.25) 2024-09-14T05:55:08.494 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-09-14T05:55:08.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-09-14T05:55:08.495 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2024-09-14T05:55:08.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-14T05:55:08.495+0000 7fca2ee17640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 36715) UID: 1000 2024-09-14T05:55:08.548 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_17.status 2024-09-14T05:55:08.548 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:55:08.548 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:08.548 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_17 2024-09-14T05:55:08.548 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_17.status 2024-09-14T05:55:08.548 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-09-14T05:55:08.548 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:55:08.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:55:08.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:55:08.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:55:08.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:55:08.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:55:08.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:55:08.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 118 = '' 2024-09-14T05:55:08.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:55:08.634 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-14T05:55:10.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_17.status 2024-09-14T05:55:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:55:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_17 2024-09-14T05:55:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_17.status 2024-09-14T05:55:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-09-14T05:55:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:55:10.291 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:55:10.292 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:55:10.293 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:55:10.293 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:55:10.294 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:55:10.294 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:55:10.294 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 118 = '' 2024-09-14T05:55:10.294 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:55:10.294 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 2 2024-09-14T05:55:10.496 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-09-14T05:55:10.496 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T05:55:10.497 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-09-14T05:55:10.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-09-14T05:55:10.497 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-09-14T05:55:10.501 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-09-14T05:55:10.501 INFO:tasks.rbd_mirror_thrash:waiting for 90 secs before thrashing 2024-09-14T05:55:12.295 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_17.status 2024-09-14T05:55:12.295 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:55:12.295 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:12.295 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_17 2024-09-14T05:55:12.295 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_17.status 2024-09-14T05:55:12.296 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-09-14T05:55:12.296 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:55:12.352 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:55:12.352 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:55:12.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:55:12.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:55:12.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:55:12.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:55:12.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 118 = '' 2024-09-14T05:55:12.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:55:12.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 2 2024-09-14T05:55:14.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_17.status 2024-09-14T05:55:14.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:55:14.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:14.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_17 2024-09-14T05:55:14.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_17.status 2024-09-14T05:55:14.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-09-14T05:55:14.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:55:14.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:55:14.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:55:14.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:55:14.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:55:14.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:55:14.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:55:14.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 118 = '' 2024-09-14T05:55:14.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:55:14.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 4 2024-09-14T05:55:18.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_17.status 2024-09-14T05:55:18.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:55:18.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:18.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_17 2024-09-14T05:55:18.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_17.status 2024-09-14T05:55:18.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-09-14T05:55:18.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:55:18.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:55:18.465 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=118 2024-09-14T05:55:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 118 = 118 2024-09-14T05:55:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:55:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_17 snap 2024-09-14T05:55:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:55:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_17 2024-09-14T05:55:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:55:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:55:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:55:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:55:19.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_17@snap 2024-09-14T05:55:19.516 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_17': 2024-09-14T05:55:19.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:55:19.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:55:19.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-14T05:55:19.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c19aa4039d 2024-09-14T05:55:19.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c19aa4039d 2024-09-14T05:55:19.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:55:19.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:55:19.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:55:19.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:55:19.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:47:08 2024 2024-09-14T05:55:19.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:47:08 2024 2024-09-14T05:55:19.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:47:08 2024 2024-09-14T05:55:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:55:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:55:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:55:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 5c00a5fc-12ce-46ba-b4c0-0e6aa090fe1a 2024-09-14T05:55:19.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:55:19.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:55:19.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_17 snap 2024-09-14T05:55:19.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:19.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_17 2024-09-14T05:55:19.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:55:19.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:55:19.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_17.export 2024-09-14T05:55:19.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_17.export 2024-09-14T05:55:19.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_17.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_17.export 2024-09-14T05:55:19.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_17@snap /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_17.export 2024-09-14T05:55:19.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:55:19.900 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_17@snap /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_17.export 2024-09-14T05:55:20.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:55:20.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_17.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_17.export 2024-09-14T05:55:20.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_17.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_17.export 2024-09-14T05:55:20.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:55:20.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:55:20.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_18 2024-09-14T05:55:20.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_18 snap 2024-09-14T05:55:20.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:55:20.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:20.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_18 2024-09-14T05:55:20.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:55:20.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_18@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T05:55:22.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_18 2024-09-14T05:55:22.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:55:22.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:22.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_18 2024-09-14T05:55:22.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_18 started 2024-09-14T05:55:22.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:55:22.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:22.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_18 2024-09-14T05:55:22.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:55:22.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:55:22.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:55:22.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:55:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_18 started 2024-09-14T05:55:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:55:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_18 2024-09-14T05:55:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:55:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:55:23.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:55:23.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_18 2024-09-14T05:55:23.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:55:23.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:55:23.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:55:23.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:55:23.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:55:23.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:55:23.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:55:23.129 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:55:23.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:55:23.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:55:23.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:55:23.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_18 2024-09-14T05:55:23.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:55:23.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:55:23.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:55:23.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:55:23.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:55:23.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:55:23.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:55:23.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:55:23.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:55:23.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:55:23.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:55:23.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:55:23.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:55:23.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:55:23.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:55:23.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:55:23.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:55:23.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:55:23.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:55:23.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:55:23.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:55:23.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:55:23.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:55:23.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:55:23.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:55:23.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:55:23.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:55:23.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:55:23.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:55:23.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:55:23.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:55:23.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:55:23.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:55:23.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_18 2024-09-14T05:55:23.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:55:23.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:55:23.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:55:23.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:55:23.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:55:23.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:55:23.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:55:23.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:55:23.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_18 2024-09-14T05:55:23.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:55:23.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:55:23.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:23.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_18 2024-09-14T05:55:23.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:55:23.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:55:23.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_18 2024-09-14T05:55:23.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:55:23.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:55:23.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:23.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_18 2024-09-14T05:55:23.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_18.status 2024-09-14T05:55:23.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_18.status 2024-09-14T05:55:23.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:55:23.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_18.status 2024-09-14T05:55:23.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_18.status 2024-09-14T05:55:23.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_18.status 2024-09-14T05:55:23.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:55:23.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_18.status 2024-09-14T05:55:23.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_18 2024-09-14T05:55:23.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:55:23.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:23.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_18 2024-09-14T05:55:23.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_18 2024-09-14T05:55:24.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 120 2024-09-14T05:55:24.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_18 /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_18.status 2024-09-14T05:55:24.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:55:24.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:24.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_18 2024-09-14T05:55:24.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_18.status 2024-09-14T05:55:24.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_18 --format xml 2024-09-14T05:55:24.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:55:24.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:55:24.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=120 2024-09-14T05:55:24.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:55:24.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:55:24.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:55:24.384 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_18.status 2024-09-14T05:55:24.384 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:55:24.384 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:24.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_18 2024-09-14T05:55:24.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_18.status 2024-09-14T05:55:24.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-09-14T05:55:24.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:55:24.436 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:55:24.437 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=52 2024-09-14T05:55:24.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 120 = 52 2024-09-14T05:55:24.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:55:24.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:55:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_18.status 2024-09-14T05:55:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:55:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_18 2024-09-14T05:55:24.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_18.status 2024-09-14T05:55:24.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-09-14T05:55:24.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:55:24.890 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:55:24.891 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=52 2024-09-14T05:55:24.891 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 120 = 52 2024-09-14T05:55:24.891 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:55:24.891 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:55:25.693 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_18.status 2024-09-14T05:55:25.693 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:55:25.693 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:25.693 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_18 2024-09-14T05:55:25.693 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_18.status 2024-09-14T05:55:25.693 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-09-14T05:55:25.693 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:55:25.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:55:25.747 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:55:25.748 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:55:25.748 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:55:25.748 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:55:25.749 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:55:25.749 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 120 = '' 2024-09-14T05:55:25.749 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:55:25.749 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-14T05:55:27.350 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_18.status 2024-09-14T05:55:27.350 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:55:27.350 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:27.350 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_18 2024-09-14T05:55:27.350 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_18.status 2024-09-14T05:55:27.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-09-14T05:55:27.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:55:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:55:27.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=120 2024-09-14T05:55:27.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 120 = 120 2024-09-14T05:55:27.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:55:27.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_18 snap 2024-09-14T05:55:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:55:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_18 2024-09-14T05:55:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:55:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:55:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:55:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:55:28.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_18@snap 2024-09-14T05:55:28.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_18': 2024-09-14T05:55:28.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:55:28.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:55:28.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-14T05:55:28.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c1fae5a14a 2024-09-14T05:55:28.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c1fae5a14a 2024-09-14T05:55:28.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:55:28.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:55:28.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:55:28.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:55:28.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:47:20 2024 2024-09-14T05:55:28.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:47:20 2024 2024-09-14T05:55:28.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:47:20 2024 2024-09-14T05:55:28.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:55:28.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:55:28.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:55:28.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 3d117385-98ae-4587-b863-f50b36e012f3 2024-09-14T05:55:28.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:55:28.465 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:55:28.465 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_18 snap 2024-09-14T05:55:28.465 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:28.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_18 2024-09-14T05:55:28.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:55:28.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:55:28.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_18.export 2024-09-14T05:55:28.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_18.export 2024-09-14T05:55:28.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_18.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_18.export 2024-09-14T05:55:28.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_18@snap /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_18.export 2024-09-14T05:55:28.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:55:28.639 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_18@snap /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_18.export 2024-09-14T05:55:28.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:55:28.859 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_18.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_18.export 2024-09-14T05:55:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_18.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_18.export 2024-09-14T05:55:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:55:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:55:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_19 2024-09-14T05:55:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_19 snap 2024-09-14T05:55:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:55:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_19 2024-09-14T05:55:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:55:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_19@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T05:55:30.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_19 2024-09-14T05:55:30.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:55:30.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:30.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_19 2024-09-14T05:55:30.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_19 started 2024-09-14T05:55:30.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:55:30.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:30.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_19 2024-09-14T05:55:30.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:55:30.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:55:30.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:55:30.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:55:31.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_19 started 2024-09-14T05:55:31.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:55:31.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:31.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_19 2024-09-14T05:55:31.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:55:31.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:55:31.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:55:31.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_19 2024-09-14T05:55:31.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:55:31.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:55:31.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:55:31.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:55:31.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:55:31.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:55:31.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:55:31.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:55:31.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:55:31.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:55:31.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:55:31.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_19 2024-09-14T05:55:31.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:55:31.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:55:31.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:55:31.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:55:31.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:55:31.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:55:31.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:55:31.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:55:31.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:55:31.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:55:31.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:55:31.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:55:31.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:55:31.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:55:31.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:55:31.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:55:31.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:55:31.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:55:31.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:55:31.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:55:31.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:55:31.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:55:31.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:55:31.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:55:31.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:55:31.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:55:31.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:55:31.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:55:31.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:55:31.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:55:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:55:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:55:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:55:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_19 2024-09-14T05:55:31.294 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:55:31.295 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:55:31.295 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:55:31.295 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:55:31.295 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:55:31.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:55:31.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:55:31.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:55:31.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_19 2024-09-14T05:55:31.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:55:31.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:55:31.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:31.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_19 2024-09-14T05:55:31.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:55:31.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:55:31.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_19 2024-09-14T05:55:31.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:55:31.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:55:31.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:31.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_19 2024-09-14T05:55:31.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_19.status 2024-09-14T05:55:31.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_19.status 2024-09-14T05:55:31.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:55:31.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_19.status 2024-09-14T05:55:31.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_19.status 2024-09-14T05:55:31.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_19.status 2024-09-14T05:55:31.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:55:31.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_19.status 2024-09-14T05:55:31.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_19 2024-09-14T05:55:31.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:55:31.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:31.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_19 2024-09-14T05:55:31.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_19 2024-09-14T05:55:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 122 2024-09-14T05:55:32.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_19 /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_19.status 2024-09-14T05:55:32.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:55:32.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:32.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_19 2024-09-14T05:55:32.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_19.status 2024-09-14T05:55:32.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_19 --format xml 2024-09-14T05:55:32.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:55:32.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:55:32.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=122 2024-09-14T05:55:32.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:55:32.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:55:32.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:55:32.398 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_19.status 2024-09-14T05:55:32.398 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:55:32.398 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:32.398 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_19 2024-09-14T05:55:32.398 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_19.status 2024-09-14T05:55:32.399 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-09-14T05:55:32.399 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:55:32.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:55:32.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=53 2024-09-14T05:55:32.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 122 = 53 2024-09-14T05:55:32.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:55:32.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:55:32.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_19.status 2024-09-14T05:55:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:55:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_19 2024-09-14T05:55:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_19.status 2024-09-14T05:55:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-09-14T05:55:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:55:32.900 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:55:32.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=53 2024-09-14T05:55:32.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 122 = 53 2024-09-14T05:55:32.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:55:32.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:55:33.703 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_19.status 2024-09-14T05:55:33.703 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:55:33.703 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:33.703 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_19 2024-09-14T05:55:33.703 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_19.status 2024-09-14T05:55:33.703 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-09-14T05:55:33.704 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:55:33.756 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:55:33.756 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:55:33.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:55:33.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:55:33.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:55:33.758 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:55:33.758 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 122 = '' 2024-09-14T05:55:33.758 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:55:33.758 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-14T05:55:34.507 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-14T05:55:34.507 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-14T05:55:34.507 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-09-14T05:55:34.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-14T05:55:34.507+0000 7fdaf04bf640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 47276) UID: 1000 2024-09-14T05:55:35.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_19.status 2024-09-14T05:55:35.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:55:35.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:35.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_19 2024-09-14T05:55:35.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_19.status 2024-09-14T05:55:35.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-09-14T05:55:35.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:55:35.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:55:35.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=122 2024-09-14T05:55:35.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 122 = 122 2024-09-14T05:55:35.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:55:35.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_19 snap 2024-09-14T05:55:35.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:55:35.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:35.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_19 2024-09-14T05:55:35.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:55:35.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:55:35.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:55:35.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:55:36.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_19@snap 2024-09-14T05:55:36.465 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_19': 2024-09-14T05:55:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:55:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:55:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-14T05:55:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c18a440e73 2024-09-14T05:55:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c18a440e73 2024-09-14T05:55:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:55:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:55:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:55:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:55:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:47:33 2024 2024-09-14T05:55:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:47:33 2024 2024-09-14T05:55:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:47:33 2024 2024-09-14T05:55:36.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:55:36.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:55:36.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:55:36.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: a166af45-2737-4c1c-912c-26014f069f63 2024-09-14T05:55:36.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:55:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:55:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_19 snap 2024-09-14T05:55:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_19 2024-09-14T05:55:36.471 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:55:36.471 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:55:36.471 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_19.export 2024-09-14T05:55:36.471 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_19.export 2024-09-14T05:55:36.471 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_19.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_19.export 2024-09-14T05:55:36.471 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_19@snap /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_19.export 2024-09-14T05:55:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:55:36.667 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_19@snap /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_19.export 2024-09-14T05:55:36.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:55:36.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_19.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_19.export 2024-09-14T05:55:36.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_19.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_19.export 2024-09-14T05:55:36.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:55:36.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:55:36.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_20 2024-09-14T05:55:36.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_20 snap 2024-09-14T05:55:36.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:55:36.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:36.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_20 2024-09-14T05:55:36.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:55:36.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_20@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T05:55:38.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_20 2024-09-14T05:55:38.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:55:38.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:38.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_20 2024-09-14T05:55:38.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_20 started 2024-09-14T05:55:38.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:55:38.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:38.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_20 2024-09-14T05:55:38.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:55:38.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:55:38.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:55:38.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:55:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_20 started 2024-09-14T05:55:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:55:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_20 2024-09-14T05:55:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:55:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:55:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:55:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_20 2024-09-14T05:55:39.158 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:55:39.158 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:55:39.158 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:55:39.158 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:55:39.158 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:55:39.158 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:55:39.158 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:55:39.158 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:55:39.158 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:55:39.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:55:39.161 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:55:39.161 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_20 2024-09-14T05:55:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:55:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:55:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:55:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:55:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:55:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:55:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:55:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:55:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:55:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:55:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:55:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:55:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:55:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:55:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:55:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:55:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:55:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:55:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:55:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:55:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:55:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:55:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:55:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:55:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:55:39.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:55:39.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:55:39.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:55:39.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:55:39.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:55:39.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:55:39.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:55:39.185 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:55:39.185 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_20 2024-09-14T05:55:39.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:55:39.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:55:39.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:55:39.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:55:39.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:55:39.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:55:39.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:55:39.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:55:39.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_20 2024-09-14T05:55:39.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:55:39.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:55:39.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:39.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_20 2024-09-14T05:55:39.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:55:39.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:55:39.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_20 2024-09-14T05:55:39.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:55:39.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:55:39.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:39.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_20 2024-09-14T05:55:39.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_20.status 2024-09-14T05:55:39.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_20.status 2024-09-14T05:55:39.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:55:39.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_20.status 2024-09-14T05:55:39.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_20.status 2024-09-14T05:55:39.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_20.status 2024-09-14T05:55:39.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:55:39.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_20.status 2024-09-14T05:55:39.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_20 2024-09-14T05:55:39.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:55:39.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:39.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_20 2024-09-14T05:55:39.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_20 2024-09-14T05:55:40.158 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 124 2024-09-14T05:55:40.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_20 /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_20.status 2024-09-14T05:55:40.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:55:40.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:40.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_20 2024-09-14T05:55:40.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_20.status 2024-09-14T05:55:40.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_20 --format xml 2024-09-14T05:55:40.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:55:40.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:55:40.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=124 2024-09-14T05:55:40.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:55:40.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:55:40.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:55:40.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_20.status 2024-09-14T05:55:40.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:55:40.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:40.421 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_20 2024-09-14T05:55:40.421 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_20.status 2024-09-14T05:55:40.421 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-09-14T05:55:40.421 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:55:40.472 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:55:40.473 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=54 2024-09-14T05:55:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 124 = 54 2024-09-14T05:55:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:55:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:55:40.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_20.status 2024-09-14T05:55:40.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:55:40.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:40.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_20 2024-09-14T05:55:40.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_20.status 2024-09-14T05:55:40.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-09-14T05:55:40.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:55:40.925 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:55:40.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=54 2024-09-14T05:55:40.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 124 = 54 2024-09-14T05:55:40.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:55:40.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:55:41.508 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-14T05:55:41.508 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T05:55:41.508 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-14T05:55:41.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-14T05:55:41.509 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-14T05:55:41.512 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-14T05:55:41.512 INFO:tasks.rbd_mirror_thrash:waiting for 85 secs before thrashing 2024-09-14T05:55:41.728 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_20.status 2024-09-14T05:55:41.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:55:41.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:41.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_20 2024-09-14T05:55:41.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_20.status 2024-09-14T05:55:41.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-09-14T05:55:41.729 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:55:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:55:41.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:55:41.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:55:41.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:55:41.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:55:41.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:55:41.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 124 = '' 2024-09-14T05:55:41.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:55:41.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-14T05:55:43.380 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_20.status 2024-09-14T05:55:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:55:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_20 2024-09-14T05:55:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_20.status 2024-09-14T05:55:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-09-14T05:55:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:55:43.435 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:55:43.437 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=124 2024-09-14T05:55:43.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 124 = 124 2024-09-14T05:55:43.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:55:43.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_20 snap 2024-09-14T05:55:43.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:55:43.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:43.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_20 2024-09-14T05:55:43.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:55:43.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:55:43.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:55:43.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:55:44.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_20@snap 2024-09-14T05:55:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_20': 2024-09-14T05:55:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:55:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:55:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-14T05:55:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c1bbf0b645 2024-09-14T05:55:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c1bbf0b645 2024-09-14T05:55:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:55:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:55:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:55:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:55:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:47:40 2024 2024-09-14T05:55:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:47:40 2024 2024-09-14T05:55:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:47:40 2024 2024-09-14T05:55:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:55:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:55:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:55:44.498 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 6654bddc-72e9-47b0-8061-615cff98334f 2024-09-14T05:55:44.498 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:55:44.500 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:55:44.500 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_20 snap 2024-09-14T05:55:44.500 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:44.500 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_20 2024-09-14T05:55:44.500 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:55:44.500 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:55:44.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_20.export 2024-09-14T05:55:44.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_20.export 2024-09-14T05:55:44.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_20.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_20.export 2024-09-14T05:55:44.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_20@snap /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_20.export 2024-09-14T05:55:44.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:55:44.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_20@snap /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_20.export 2024-09-14T05:55:44.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:55:44.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_20.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_20.export 2024-09-14T05:55:44.931 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_20.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_20.export 2024-09-14T05:55:44.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:55:44.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:55:44.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_21 2024-09-14T05:55:44.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_21 snap 2024-09-14T05:55:44.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:55:44.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:44.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_21 2024-09-14T05:55:44.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:55:44.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_21@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T05:55:46.186 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_21 2024-09-14T05:55:46.186 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:55:46.186 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:46.186 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_21 2024-09-14T05:55:46.186 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_21 started 2024-09-14T05:55:46.186 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:55:46.187 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:46.187 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_21 2024-09-14T05:55:46.187 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:55:46.187 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:55:46.187 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:55:46.187 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:55:47.187 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_21 started 2024-09-14T05:55:47.187 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:55:47.187 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:47.187 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_21 2024-09-14T05:55:47.187 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:55:47.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:55:47.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:55:47.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_21 2024-09-14T05:55:47.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:55:47.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:55:47.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:55:47.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:55:47.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:55:47.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:55:47.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:55:47.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:55:47.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:55:47.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:55:47.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:55:47.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_21 2024-09-14T05:55:47.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:55:47.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:55:47.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:55:47.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:55:47.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:55:47.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:55:47.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:55:47.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:55:47.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:55:47.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:55:47.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:55:47.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:55:47.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:55:47.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:55:47.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:55:47.193 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:55:47.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:55:47.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:55:47.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:55:47.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:55:47.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:55:47.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:55:47.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:55:47.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:55:47.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:55:47.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:55:47.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:55:47.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:55:47.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:55:47.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:55:47.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:55:47.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:55:47.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:55:47.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_21 2024-09-14T05:55:47.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:55:47.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:55:47.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:55:47.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:55:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:55:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:55:47.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:55:47.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:55:47.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_21 2024-09-14T05:55:47.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:55:47.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:55:47.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:47.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_21 2024-09-14T05:55:47.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:55:47.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:55:47.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_21 2024-09-14T05:55:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:55:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:55:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_21 2024-09-14T05:55:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_21.status 2024-09-14T05:55:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_21.status 2024-09-14T05:55:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:55:47.340 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_21.status 2024-09-14T05:55:47.340 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_21.status 2024-09-14T05:55:47.340 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_21.status 2024-09-14T05:55:47.341 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:55:47.341 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_21.status 2024-09-14T05:55:47.341 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_21 2024-09-14T05:55:47.341 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:55:47.341 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:47.341 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_21 2024-09-14T05:55:47.341 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_21 2024-09-14T05:55:48.185 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 126 2024-09-14T05:55:48.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_21 /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_21.status 2024-09-14T05:55:48.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:55:48.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:48.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_21 2024-09-14T05:55:48.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_21.status 2024-09-14T05:55:48.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_21 --format xml 2024-09-14T05:55:48.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:55:48.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:55:48.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=126 2024-09-14T05:55:48.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:55:48.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:55:48.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:55:48.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_21.status 2024-09-14T05:55:48.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:55:48.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:48.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_21 2024-09-14T05:55:48.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_21.status 2024-09-14T05:55:48.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-09-14T05:55:48.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:55:48.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:55:48.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=55 2024-09-14T05:55:48.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 126 = 55 2024-09-14T05:55:48.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:55:48.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:55:48.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_21.status 2024-09-14T05:55:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:55:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_21 2024-09-14T05:55:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_21.status 2024-09-14T05:55:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-09-14T05:55:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:55:48.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:55:48.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=55 2024-09-14T05:55:48.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 126 = 55 2024-09-14T05:55:48.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:55:48.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:55:49.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_21.status 2024-09-14T05:55:49.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:55:49.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:49.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_21 2024-09-14T05:55:49.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_21.status 2024-09-14T05:55:49.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-09-14T05:55:49.760 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:55:49.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:55:49.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:55:49.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:55:49.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:55:49.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:55:49.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:55:49.821 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 126 = '' 2024-09-14T05:55:49.821 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:55:49.821 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-14T05:55:51.422 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_21.status 2024-09-14T05:55:51.422 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:55:51.422 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:51.422 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_21 2024-09-14T05:55:51.422 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_21.status 2024-09-14T05:55:51.422 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-09-14T05:55:51.422 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:55:51.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:55:51.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=126 2024-09-14T05:55:51.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 126 = 126 2024-09-14T05:55:51.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:55:51.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_21 snap 2024-09-14T05:55:51.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:55:51.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:51.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_21 2024-09-14T05:55:51.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:55:51.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:55:51.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:55:51.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:55:52.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_21@snap 2024-09-14T05:55:52.529 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_21': 2024-09-14T05:55:52.529 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:55:52.529 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:55:52.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-14T05:55:52.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c119d20673 2024-09-14T05:55:52.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c119d20673 2024-09-14T05:55:52.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:55:52.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:55:52.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:55:52.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:55:52.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:47:48 2024 2024-09-14T05:55:52.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:47:48 2024 2024-09-14T05:55:52.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:47:48 2024 2024-09-14T05:55:52.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:55:52.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:55:52.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:55:52.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: a03aba9f-8291-447c-bf72-b41385e7c989 2024-09-14T05:55:52.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:55:52.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:55:52.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_21 snap 2024-09-14T05:55:52.534 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:52.534 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_21 2024-09-14T05:55:52.534 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:55:52.534 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:55:52.534 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_21.export 2024-09-14T05:55:52.534 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_21.export 2024-09-14T05:55:52.534 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_21.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_21.export 2024-09-14T05:55:52.534 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_21@snap /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_21.export 2024-09-14T05:55:52.709 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:55:52.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_21@snap /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_21.export 2024-09-14T05:55:52.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:55:52.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_21.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_21.export 2024-09-14T05:55:52.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_21.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_21.export 2024-09-14T05:55:53.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:55:53.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:55:53.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_22 2024-09-14T05:55:53.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_22 snap 2024-09-14T05:55:53.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:55:53.014 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:53.014 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_22 2024-09-14T05:55:53.014 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:55:53.014 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_22@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T05:55:54.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_22 2024-09-14T05:55:54.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:55:54.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:54.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_22 2024-09-14T05:55:54.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_22 started 2024-09-14T05:55:54.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:55:54.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:54.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_22 2024-09-14T05:55:54.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:55:54.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:55:54.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:55:54.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:55:55.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_22 started 2024-09-14T05:55:55.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:55:55.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:55.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_22 2024-09-14T05:55:55.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:55:55.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:55:55.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:55:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_22 2024-09-14T05:55:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:55:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:55:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:55:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:55:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:55:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:55:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:55:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:55:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:55:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:55:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:55:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_22 2024-09-14T05:55:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:55:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:55:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:55:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:55:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:55:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:55:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:55:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:55:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:55:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:55:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:55:55.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:55:55.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:55:55.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:55:55.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:55:55.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:55:55.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:55:55.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:55:55.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:55:55.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:55:55.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:55:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:55:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:55:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:55:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:55:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:55:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:55:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:55:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:55:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:55:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:55:55.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:55:55.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:55:55.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_22 2024-09-14T05:55:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:55:55.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:55:55.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:55:55.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:55:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:55:55.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:55:55.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:55:55.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:55:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_22 2024-09-14T05:55:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:55:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:55:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_22 2024-09-14T05:55:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:55:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:55:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_22 2024-09-14T05:55:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:55:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:55:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_22 2024-09-14T05:55:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_22.status 2024-09-14T05:55:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_22.status 2024-09-14T05:55:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:55:55.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_22.status 2024-09-14T05:55:55.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_22.status 2024-09-14T05:55:55.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_22.status 2024-09-14T05:55:55.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:55:55.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_22.status 2024-09-14T05:55:55.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_22 2024-09-14T05:55:55.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:55:55.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:55.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_22 2024-09-14T05:55:55.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_22 2024-09-14T05:55:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 128 2024-09-14T05:55:56.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_22 /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_22.status 2024-09-14T05:55:56.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:55:56.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:56.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_22 2024-09-14T05:55:56.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_22.status 2024-09-14T05:55:56.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_22 --format xml 2024-09-14T05:55:56.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:55:56.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:55:56.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=128 2024-09-14T05:55:56.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:55:56.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:55:56.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:55:56.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_22.status 2024-09-14T05:55:56.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:55:56.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:56.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_22 2024-09-14T05:55:56.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_22.status 2024-09-14T05:55:56.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-09-14T05:55:56.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:55:56.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:55:56.514 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=56 2024-09-14T05:55:56.514 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 128 = 56 2024-09-14T05:55:56.514 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:55:56.514 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:55:56.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_22.status 2024-09-14T05:55:56.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:55:56.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:56.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_22 2024-09-14T05:55:56.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_22.status 2024-09-14T05:55:56.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-09-14T05:55:56.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:55:56.966 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:55:56.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=56 2024-09-14T05:55:56.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 128 = 56 2024-09-14T05:55:56.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:55:56.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:55:57.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_22.status 2024-09-14T05:55:57.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:55:57.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:57.769 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_22 2024-09-14T05:55:57.770 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_22.status 2024-09-14T05:55:57.770 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-09-14T05:55:57.770 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:55:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:55:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:55:57.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:55:57.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:55:57.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:55:57.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:55:57.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 128 = '' 2024-09-14T05:55:57.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:55:57.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-14T05:55:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_22.status 2024-09-14T05:55:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:55:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_22 2024-09-14T05:55:59.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_22.status 2024-09-14T05:55:59.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-09-14T05:55:59.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:55:59.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:55:59.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=128 2024-09-14T05:55:59.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 128 = 128 2024-09-14T05:55:59.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:55:59.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_22 snap 2024-09-14T05:55:59.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:55:59.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:55:59.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_22 2024-09-14T05:55:59.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:55:59.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:55:59.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:55:59.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:56:00.489 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_22@snap 2024-09-14T05:56:00.538 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_22': 2024-09-14T05:56:00.538 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:56:00.538 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:56:00.538 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-14T05:56:00.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c1a8cb3204 2024-09-14T05:56:00.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c1a8cb3204 2024-09-14T05:56:00.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:56:00.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:56:00.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:56:00.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:56:00.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:47:58 2024 2024-09-14T05:56:00.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:47:58 2024 2024-09-14T05:56:00.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:47:58 2024 2024-09-14T05:56:00.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:56:00.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:56:00.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:56:00.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 81e2e849-43ea-4a3d-825a-ccda8c67058a 2024-09-14T05:56:00.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:56:00.542 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:56:00.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_22 snap 2024-09-14T05:56:00.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:00.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_22 2024-09-14T05:56:00.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:56:00.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:56:00.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_22.export 2024-09-14T05:56:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_22.export 2024-09-14T05:56:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_22.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_22.export 2024-09-14T05:56:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_22@snap /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_22.export 2024-09-14T05:56:00.719 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:56:00.728 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_22@snap /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_22.export 2024-09-14T05:56:00.929 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:56:00.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_22.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_22.export 2024-09-14T05:56:00.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_22.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_22.export 2024-09-14T05:56:01.021 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:56:01.022 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:56:01.022 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_23 2024-09-14T05:56:01.022 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_23 snap 2024-09-14T05:56:01.022 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:56:01.022 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:01.022 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_23 2024-09-14T05:56:01.022 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:56:01.022 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_23@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T05:56:02.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_23 2024-09-14T05:56:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_23 2024-09-14T05:56:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_23 started 2024-09-14T05:56:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_23 2024-09-14T05:56:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:56:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:56:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:56:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:56:03.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_23 started 2024-09-14T05:56:03.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:03.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:03.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_23 2024-09-14T05:56:03.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:56:03.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:56:03.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:56:03.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_23 2024-09-14T05:56:03.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:56:03.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:56:03.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:56:03.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:56:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:56:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:56:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:56:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:56:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:56:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:56:03.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:56:03.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_23 2024-09-14T05:56:03.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:56:03.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:56:03.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:56:03.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:56:03.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:56:03.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:56:03.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:56:03.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:56:03.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:56:03.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:56:03.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:56:03.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:56:03.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:56:03.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:56:03.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:56:03.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:56:03.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:56:03.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:56:03.811 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:56:03.811 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:56:03.811 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:56:03.811 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:56:03.811 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:56:03.811 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:56:03.812 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:56:03.812 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:56:03.812 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:56:03.812 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:56:03.812 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:56:03.812 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:56:03.830 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:56:03.830 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:56:03.830 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:56:03.830 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_23 2024-09-14T05:56:03.947 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:56:03.947 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:56:03.947 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:56:03.947 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:56:03.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:56:03.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:56:03.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:56:03.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:56:03.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_23 2024-09-14T05:56:03.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:56:03.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:56:03.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:03.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_23 2024-09-14T05:56:03.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:56:03.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:56:03.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_23 2024-09-14T05:56:03.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:56:03.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:56:03.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:03.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_23 2024-09-14T05:56:03.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_23.status 2024-09-14T05:56:03.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_23.status 2024-09-14T05:56:03.951 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:56:03.952 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_23.status 2024-09-14T05:56:03.952 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_23.status 2024-09-14T05:56:03.952 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_23.status 2024-09-14T05:56:03.952 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:56:03.952 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_23.status 2024-09-14T05:56:03.952 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_23 2024-09-14T05:56:03.952 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:56:03.952 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:03.952 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_23 2024-09-14T05:56:03.952 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_23 2024-09-14T05:56:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 130 2024-09-14T05:56:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_23 /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_23.status 2024-09-14T05:56:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:56:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_23 2024-09-14T05:56:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_23.status 2024-09-14T05:56:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_23 --format xml 2024-09-14T05:56:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:56:04.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:56:04.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=130 2024-09-14T05:56:04.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:56:04.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:56:04.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:56:05.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_23.status 2024-09-14T05:56:05.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:05.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:05.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_23 2024-09-14T05:56:05.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_23.status 2024-09-14T05:56:05.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-09-14T05:56:05.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:56:05.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:56:05.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:56:05.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:56:05.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:56:05.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:56:05.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:56:05.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 130 = '' 2024-09-14T05:56:05.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:56:05.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:56:05.528 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_23.status 2024-09-14T05:56:05.528 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:05.529 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:05.529 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_23 2024-09-14T05:56:05.529 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_23.status 2024-09-14T05:56:05.529 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-09-14T05:56:05.529 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:56:05.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:56:05.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:56:05.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:56:05.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:56:05.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:56:05.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:56:05.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 130 = '' 2024-09-14T05:56:05.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:56:05.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:56:06.383 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_23.status 2024-09-14T05:56:06.383 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:06.383 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:06.383 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_23 2024-09-14T05:56:06.383 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_23.status 2024-09-14T05:56:06.383 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-09-14T05:56:06.384 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:56:06.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:56:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=130 2024-09-14T05:56:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 130 = 130 2024-09-14T05:56:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:56:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_23 snap 2024-09-14T05:56:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_23 2024-09-14T05:56:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:56:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:56:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:56:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:56:07.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_23@snap 2024-09-14T05:56:07.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_23': 2024-09-14T05:56:07.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:56:07.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:56:07.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-14T05:56:07.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c17d44e3b 2024-09-14T05:56:07.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c17d44e3b 2024-09-14T05:56:07.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:56:07.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:56:07.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:56:07.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:56:07.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:48:12 2024 2024-09-14T05:56:07.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:48:12 2024 2024-09-14T05:56:07.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:48:12 2024 2024-09-14T05:56:07.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:56:07.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:56:07.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:56:07.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 1ad775b2-4acb-4f43-b70c-7d7c1bb4782d 2024-09-14T05:56:07.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:56:07.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:56:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_23 snap 2024-09-14T05:56:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_23 2024-09-14T05:56:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:56:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:56:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_23.export 2024-09-14T05:56:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_23.export 2024-09-14T05:56:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_23.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_23.export 2024-09-14T05:56:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_23@snap /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_23.export 2024-09-14T05:56:07.653 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:56:07.660 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_23@snap /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_23.export 2024-09-14T05:56:07.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:56:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_23.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_23.export 2024-09-14T05:56:07.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_23.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_23.export 2024-09-14T05:56:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:56:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:56:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_24 2024-09-14T05:56:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_24 snap 2024-09-14T05:56:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:56:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_24 2024-09-14T05:56:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:56:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_24@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T05:56:09.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_24 2024-09-14T05:56:09.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:09.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:09.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_24 2024-09-14T05:56:09.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_24 started 2024-09-14T05:56:09.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:09.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:09.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_24 2024-09-14T05:56:09.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:56:09.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:56:09.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:56:09.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:56:10.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_24 started 2024-09-14T05:56:10.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:10.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:10.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_24 2024-09-14T05:56:10.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:56:10.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:56:10.834 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:56:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_24 2024-09-14T05:56:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:56:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:56:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:56:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:56:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:56:10.836 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:56:10.836 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:56:10.836 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:56:10.836 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:56:10.836 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:56:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:56:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_24 2024-09-14T05:56:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:56:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:56:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:56:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:56:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:56:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:56:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:56:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:56:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:56:10.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:56:10.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:56:10.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:56:10.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:56:10.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:56:10.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:56:10.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:56:10.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:56:10.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:56:10.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:56:10.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:56:10.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:56:10.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:56:10.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:56:10.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:56:10.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:56:10.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:56:10.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:56:10.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:56:10.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:56:10.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:56:10.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:56:10.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:56:10.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:56:10.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_24 2024-09-14T05:56:10.983 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:56:10.983 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:56:10.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:56:10.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:56:10.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:56:10.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:56:10.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:56:10.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:56:10.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_24 2024-09-14T05:56:10.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:56:10.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:56:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_24 2024-09-14T05:56:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:56:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:56:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_24 2024-09-14T05:56:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:56:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:56:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_24 2024-09-14T05:56:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_24.status 2024-09-14T05:56:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_24.status 2024-09-14T05:56:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:56:10.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_24.status 2024-09-14T05:56:10.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_24.status 2024-09-14T05:56:10.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_24.status 2024-09-14T05:56:10.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:56:10.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_24.status 2024-09-14T05:56:10.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_24 2024-09-14T05:56:10.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:56:10.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:10.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_24 2024-09-14T05:56:10.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_24 2024-09-14T05:56:11.831 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 132 2024-09-14T05:56:11.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_24 /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_24.status 2024-09-14T05:56:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:56:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_24 2024-09-14T05:56:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_24.status 2024-09-14T05:56:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_24 --format xml 2024-09-14T05:56:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:56:11.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:56:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=132 2024-09-14T05:56:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:56:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:56:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:56:12.091 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_24.status 2024-09-14T05:56:12.091 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:12.091 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:12.091 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_24 2024-09-14T05:56:12.091 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_24.status 2024-09-14T05:56:12.091 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-09-14T05:56:12.091 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:56:12.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:56:12.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:56:12.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:56:12.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:56:12.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:56:12.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:56:12.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 132 = '' 2024-09-14T05:56:12.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:56:12.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:56:12.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_24.status 2024-09-14T05:56:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_24 2024-09-14T05:56:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_24.status 2024-09-14T05:56:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-09-14T05:56:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:56:12.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:56:12.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:56:12.595 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:56:12.595 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:56:12.595 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:56:12.595 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:56:12.595 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 132 = '' 2024-09-14T05:56:12.595 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:56:12.595 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:56:13.396 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_24.status 2024-09-14T05:56:13.396 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:13.396 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:13.396 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_24 2024-09-14T05:56:13.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_24.status 2024-09-14T05:56:13.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-09-14T05:56:13.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:56:13.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:56:13.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=132 2024-09-14T05:56:13.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 132 = 132 2024-09-14T05:56:13.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:56:13.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_24 snap 2024-09-14T05:56:13.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:13.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:13.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_24 2024-09-14T05:56:13.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:56:13.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:56:13.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:56:13.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:56:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_24@snap 2024-09-14T05:56:14.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_24': 2024-09-14T05:56:14.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:56:14.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:56:14.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-14T05:56:14.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c1201ffdcf 2024-09-14T05:56:14.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c1201ffdcf 2024-09-14T05:56:14.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:56:14.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:56:14.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:56:14.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:56:14.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:48:18 2024 2024-09-14T05:56:14.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:48:18 2024 2024-09-14T05:56:14.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:48:18 2024 2024-09-14T05:56:14.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:56:14.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:56:14.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:56:14.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 0470f29c-7ca8-419d-8a53-5b2f3f342a4f 2024-09-14T05:56:14.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:56:14.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:56:14.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_24 snap 2024-09-14T05:56:14.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:14.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_24 2024-09-14T05:56:14.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:56:14.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:56:14.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_24.export 2024-09-14T05:56:14.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_24.export 2024-09-14T05:56:14.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_24.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_24.export 2024-09-14T05:56:14.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_24@snap /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_24.export 2024-09-14T05:56:14.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:56:14.686 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_24@snap /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_24.export 2024-09-14T05:56:14.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:56:14.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_24.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_24.export 2024-09-14T05:56:14.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_24.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_24.export 2024-09-14T05:56:14.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:56:14.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:56:14.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_25 2024-09-14T05:56:14.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_25 snap 2024-09-14T05:56:14.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:56:14.960 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:14.960 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_25 2024-09-14T05:56:14.960 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:56:14.960 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_25@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T05:56:16.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_25 2024-09-14T05:56:16.844 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:16.844 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:16.844 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_25 2024-09-14T05:56:16.844 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_25 started 2024-09-14T05:56:16.844 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:16.844 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:16.844 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_25 2024-09-14T05:56:16.844 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:56:16.844 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:56:16.844 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:56:16.844 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:56:17.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_25 started 2024-09-14T05:56:17.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:17.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:17.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_25 2024-09-14T05:56:17.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:56:17.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:56:17.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:56:17.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_25 2024-09-14T05:56:17.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:56:17.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:56:17.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:56:17.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:56:17.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:56:17.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:56:17.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:56:17.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:56:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:56:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:56:17.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:56:17.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_25 2024-09-14T05:56:17.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:56:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:56:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:56:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:56:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:56:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:56:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:56:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:56:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:56:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:56:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:56:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:56:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:56:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:56:17.851 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:56:17.851 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:56:17.852 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:56:17.852 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:56:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:56:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:56:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:56:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:56:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:56:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:56:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:56:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:56:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:56:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:56:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:56:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:56:17.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:56:17.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:56:17.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:56:17.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_25 2024-09-14T05:56:17.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:56:17.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:56:17.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:56:17.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:56:17.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:56:17.991 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:56:17.991 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:56:17.991 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:56:17.991 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_25 2024-09-14T05:56:17.991 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:56:17.991 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:56:17.991 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:17.991 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_25 2024-09-14T05:56:17.991 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:56:17.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:56:17.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_25 2024-09-14T05:56:17.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:56:17.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:56:17.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:17.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_25 2024-09-14T05:56:17.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_25.status 2024-09-14T05:56:17.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_25.status 2024-09-14T05:56:17.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:56:17.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_25.status 2024-09-14T05:56:17.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_25.status 2024-09-14T05:56:17.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_25.status 2024-09-14T05:56:17.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:56:17.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_25.status 2024-09-14T05:56:17.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_25 2024-09-14T05:56:17.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:56:17.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:17.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_25 2024-09-14T05:56:17.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_25 2024-09-14T05:56:18.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 134 2024-09-14T05:56:18.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_25 /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_25.status 2024-09-14T05:56:18.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:56:18.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:18.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_25 2024-09-14T05:56:18.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_25.status 2024-09-14T05:56:18.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_25 --format xml 2024-09-14T05:56:18.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:56:18.923 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:56:18.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=134 2024-09-14T05:56:18.925 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:56:18.925 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:56:18.925 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:56:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_25.status 2024-09-14T05:56:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_25 2024-09-14T05:56:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_25.status 2024-09-14T05:56:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-09-14T05:56:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:56:19.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:56:19.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:56:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:56:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:56:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:56:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:56:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 134 = '' 2024-09-14T05:56:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:56:19.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:56:19.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_25.status 2024-09-14T05:56:19.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:19.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:19.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_25 2024-09-14T05:56:19.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_25.status 2024-09-14T05:56:19.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-09-14T05:56:19.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:56:19.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:56:19.631 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:56:19.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:56:19.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:56:19.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:56:19.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:56:19.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 134 = '' 2024-09-14T05:56:19.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:56:19.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:56:20.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_25.status 2024-09-14T05:56:20.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:20.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:20.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_25 2024-09-14T05:56:20.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_25.status 2024-09-14T05:56:20.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-09-14T05:56:20.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:56:20.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:56:20.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=134 2024-09-14T05:56:20.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 134 = 134 2024-09-14T05:56:20.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:56:20.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_25 snap 2024-09-14T05:56:20.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_25 2024-09-14T05:56:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:56:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:56:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:56:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:56:21.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_25@snap 2024-09-14T05:56:21.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_25': 2024-09-14T05:56:21.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:56:21.532 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:56:21.532 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-14T05:56:21.532 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c19cc3cef7 2024-09-14T05:56:21.532 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c19cc3cef7 2024-09-14T05:56:21.532 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:56:21.532 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:56:21.532 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:56:21.532 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:56:21.532 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:48:30 2024 2024-09-14T05:56:21.532 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:48:30 2024 2024-09-14T05:56:21.532 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:48:30 2024 2024-09-14T05:56:21.532 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:56:21.532 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:56:21.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:56:21.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: a6b55bc2-53d1-494f-9e17-76231088f41c 2024-09-14T05:56:21.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:56:21.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:56:21.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_25 snap 2024-09-14T05:56:21.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:21.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_25 2024-09-14T05:56:21.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:56:21.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:56:21.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_25.export 2024-09-14T05:56:21.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_25.export 2024-09-14T05:56:21.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_25.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_25.export 2024-09-14T05:56:21.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_25@snap /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_25.export 2024-09-14T05:56:21.698 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:56:21.705 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_25@snap /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_25.export 2024-09-14T05:56:21.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:56:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_25.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_25.export 2024-09-14T05:56:22.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_25.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_25.export 2024-09-14T05:56:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:56:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:56:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_26 2024-09-14T05:56:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_26 snap 2024-09-14T05:56:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:56:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_26 2024-09-14T05:56:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:56:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_26@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T05:56:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_26 2024-09-14T05:56:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_26 2024-09-14T05:56:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_26 started 2024-09-14T05:56:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:23.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_26 2024-09-14T05:56:23.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:56:23.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:56:23.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:56:23.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:56:24.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_26 started 2024-09-14T05:56:24.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:24.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:24.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_26 2024-09-14T05:56:24.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:56:24.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:56:24.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:56:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_26 2024-09-14T05:56:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:56:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:56:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:56:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:56:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:56:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:56:24.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:56:24.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:56:24.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:56:24.871 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:56:24.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:56:24.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_26 2024-09-14T05:56:24.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:56:24.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:56:24.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:56:24.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:56:24.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:56:24.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:56:24.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:56:24.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:56:24.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:56:24.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:56:24.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:56:24.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:56:24.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:56:24.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:56:24.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:56:24.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:56:24.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:56:24.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:56:24.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:56:24.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:56:24.874 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:56:24.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:56:24.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:56:24.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:56:24.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:56:24.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:56:24.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:56:24.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:56:24.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:56:24.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:56:24.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:56:24.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:56:24.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:56:24.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_26 2024-09-14T05:56:25.014 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:56:25.014 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:56:25.014 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:56:25.014 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:56:25.015 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:56:25.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:56:25.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:56:25.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:56:25.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_26 2024-09-14T05:56:25.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:56:25.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:56:25.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:25.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_26 2024-09-14T05:56:25.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:56:25.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:56:25.018 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_26 2024-09-14T05:56:25.018 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:56:25.018 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:56:25.018 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:25.018 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_26 2024-09-14T05:56:25.018 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_26.status 2024-09-14T05:56:25.018 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_26.status 2024-09-14T05:56:25.018 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:56:25.019 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_26.status 2024-09-14T05:56:25.019 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_26.status 2024-09-14T05:56:25.019 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_26.status 2024-09-14T05:56:25.019 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:56:25.019 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_26.status 2024-09-14T05:56:25.020 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_26 2024-09-14T05:56:25.020 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:56:25.020 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:25.020 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_26 2024-09-14T05:56:25.020 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_26 2024-09-14T05:56:25.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 136 2024-09-14T05:56:25.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_26 /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_26.status 2024-09-14T05:56:25.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:56:25.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:25.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_26 2024-09-14T05:56:25.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_26.status 2024-09-14T05:56:25.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_26 --format xml 2024-09-14T05:56:25.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:56:25.930 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:56:25.931 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=136 2024-09-14T05:56:25.931 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:56:25.931 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:56:25.932 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:56:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_26.status 2024-09-14T05:56:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_26 2024-09-14T05:56:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_26.status 2024-09-14T05:56:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-09-14T05:56:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:56:26.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:56:26.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:56:26.185 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:56:26.185 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:56:26.185 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:56:26.185 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:56:26.185 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 136 = '' 2024-09-14T05:56:26.185 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:56:26.185 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:56:26.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_26.status 2024-09-14T05:56:26.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:26.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:26.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_26 2024-09-14T05:56:26.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_26.status 2024-09-14T05:56:26.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-09-14T05:56:26.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:56:26.638 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:56:26.639 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:56:26.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:56:26.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:56:26.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:56:26.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:56:26.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 136 = '' 2024-09-14T05:56:26.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:56:26.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:56:27.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_26.status 2024-09-14T05:56:27.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:27.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:27.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_26 2024-09-14T05:56:27.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_26.status 2024-09-14T05:56:27.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-09-14T05:56:27.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:56:27.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:56:27.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=136 2024-09-14T05:56:27.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 136 = 136 2024-09-14T05:56:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:56:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_26 snap 2024-09-14T05:56:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_26 2024-09-14T05:56:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:56:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:56:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:56:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:56:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_26@snap 2024-09-14T05:56:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_26': 2024-09-14T05:56:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:56:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:56:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-14T05:56:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c161383a7b 2024-09-14T05:56:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c161383a7b 2024-09-14T05:56:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:56:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:56:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:56:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:56:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:48:40 2024 2024-09-14T05:56:28.542 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:48:40 2024 2024-09-14T05:56:28.542 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:48:40 2024 2024-09-14T05:56:28.542 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:56:28.542 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:56:28.542 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:56:28.542 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 9154d447-2cac-4f96-97e0-0ce2dbafe588 2024-09-14T05:56:28.542 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:56:28.546 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:56:28.546 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_26 snap 2024-09-14T05:56:28.546 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:28.546 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_26 2024-09-14T05:56:28.546 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:56:28.546 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:56:28.546 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_26.export 2024-09-14T05:56:28.546 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_26.export 2024-09-14T05:56:28.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_26.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_26.export 2024-09-14T05:56:28.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_26@snap /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_26.export 2024-09-14T05:56:28.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:56:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_26@snap /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_26.export 2024-09-14T05:56:28.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:56:28.965 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_26.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_26.export 2024-09-14T05:56:29.020 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_26.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_26.export 2024-09-14T05:56:29.051 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:56:29.051 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:56:29.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_27 2024-09-14T05:56:29.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_27 snap 2024-09-14T05:56:29.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:56:29.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:29.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_27 2024-09-14T05:56:29.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:56:29.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_27@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T05:56:30.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_27 2024-09-14T05:56:30.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:30.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:30.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_27 2024-09-14T05:56:30.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_27 started 2024-09-14T05:56:30.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:30.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:30.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_27 2024-09-14T05:56:30.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:56:30.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:56:30.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:56:30.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:56:31.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_27 started 2024-09-14T05:56:31.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:31.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:31.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_27 2024-09-14T05:56:31.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:56:31.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:56:31.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:56:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_27 2024-09-14T05:56:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:56:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:56:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:56:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:56:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:56:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:56:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:56:31.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:56:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:56:31.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:56:31.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:56:31.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_27 2024-09-14T05:56:31.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:56:31.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:56:31.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:56:31.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:56:31.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:56:31.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:56:31.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:56:31.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:56:31.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:56:31.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:56:31.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:56:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:56:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:56:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:56:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:56:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:56:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:56:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:56:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:56:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:56:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:56:31.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:56:31.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:56:31.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:56:31.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:56:31.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:56:31.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:56:31.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:56:31.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:56:31.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:56:31.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:56:31.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:56:31.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:56:31.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_27 2024-09-14T05:56:32.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:56:32.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:56:32.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:56:32.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:56:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:56:32.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:56:32.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:56:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:56:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_27 2024-09-14T05:56:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:56:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:56:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_27 2024-09-14T05:56:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:56:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:56:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_27 2024-09-14T05:56:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:56:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:56:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_27 2024-09-14T05:56:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_27.status 2024-09-14T05:56:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_27.status 2024-09-14T05:56:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:56:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_27.status 2024-09-14T05:56:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_27.status 2024-09-14T05:56:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_27.status 2024-09-14T05:56:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:56:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_27.status 2024-09-14T05:56:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_27 2024-09-14T05:56:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:56:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_27 2024-09-14T05:56:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_27 2024-09-14T05:56:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 138 2024-09-14T05:56:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_27 /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_27.status 2024-09-14T05:56:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:56:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_27 2024-09-14T05:56:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_27.status 2024-09-14T05:56:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_27 --format xml 2024-09-14T05:56:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:56:32.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:56:32.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=138 2024-09-14T05:56:32.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:56:32.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:56:32.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:56:33.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_27.status 2024-09-14T05:56:33.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:33.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:33.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_27 2024-09-14T05:56:33.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_27.status 2024-09-14T05:56:33.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-09-14T05:56:33.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:56:33.198 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:56:33.198 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:56:33.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:56:33.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:56:33.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:56:33.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:56:33.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 138 = '' 2024-09-14T05:56:33.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:56:33.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:56:33.601 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_27.status 2024-09-14T05:56:33.601 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:33.601 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_27 2024-09-14T05:56:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_27.status 2024-09-14T05:56:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-09-14T05:56:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:56:33.654 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:56:33.655 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:56:33.656 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:56:33.656 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:56:33.656 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:56:33.657 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:56:33.657 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 138 = '' 2024-09-14T05:56:33.657 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:56:33.657 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:56:34.458 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_27.status 2024-09-14T05:56:34.458 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:34.459 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:34.459 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_27 2024-09-14T05:56:34.459 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_27.status 2024-09-14T05:56:34.459 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-09-14T05:56:34.459 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:56:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:56:34.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=138 2024-09-14T05:56:34.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 138 = 138 2024-09-14T05:56:34.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:56:34.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_27 snap 2024-09-14T05:56:34.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:34.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:34.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_27 2024-09-14T05:56:34.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:56:34.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:56:34.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:56:34.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:56:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_27@snap 2024-09-14T05:56:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_27': 2024-09-14T05:56:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:56:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:56:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-14T05:56:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c14b024b1e 2024-09-14T05:56:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c14b024b1e 2024-09-14T05:56:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:56:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:56:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:56:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:56:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:48:54 2024 2024-09-14T05:56:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:48:54 2024 2024-09-14T05:56:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:48:54 2024 2024-09-14T05:56:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:56:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:56:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:56:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: c4de874f-fa8c-4df7-a048-8394cb50d034 2024-09-14T05:56:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:56:35.563 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:56:35.563 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_27 snap 2024-09-14T05:56:35.564 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:35.564 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_27 2024-09-14T05:56:35.564 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:56:35.564 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:56:35.564 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_27.export 2024-09-14T05:56:35.564 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_27.export 2024-09-14T05:56:35.564 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_27.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_27.export 2024-09-14T05:56:35.565 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_27@snap /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_27.export 2024-09-14T05:56:35.748 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:56:35.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_27@snap /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_27.export 2024-09-14T05:56:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:56:35.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_27.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_27.export 2024-09-14T05:56:36.041 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_27.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_27.export 2024-09-14T05:56:36.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:56:36.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:56:36.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_28 2024-09-14T05:56:36.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_28 snap 2024-09-14T05:56:36.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:56:36.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:36.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_28 2024-09-14T05:56:36.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:56:36.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_28@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T05:56:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_28 2024-09-14T05:56:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_28 2024-09-14T05:56:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_28 started 2024-09-14T05:56:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_28 2024-09-14T05:56:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:56:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:56:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:56:37.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:56:38.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_28 started 2024-09-14T05:56:38.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:38.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:38.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_28 2024-09-14T05:56:38.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:56:38.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:56:38.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:56:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_28 2024-09-14T05:56:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:56:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:56:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:56:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:56:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:56:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:56:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:56:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:56:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:56:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:56:38.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:56:38.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_28 2024-09-14T05:56:38.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:56:38.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:56:38.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:56:38.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:56:38.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:56:38.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:56:38.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:56:38.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:56:38.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:56:38.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:56:38.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:56:38.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:56:38.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:56:38.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:56:38.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:56:38.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:56:38.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:56:38.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:56:38.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:56:38.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:56:38.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:56:38.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:56:38.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:56:38.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:56:38.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:56:38.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:56:38.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:56:38.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:56:38.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:56:38.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:56:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:56:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:56:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:56:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_28 2024-09-14T05:56:39.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:56:39.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:56:39.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:56:39.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:56:39.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:56:39.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:56:39.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:56:39.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:56:39.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_28 2024-09-14T05:56:39.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:56:39.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:56:39.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:39.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_28 2024-09-14T05:56:39.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:56:39.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:56:39.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_28 2024-09-14T05:56:39.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:56:39.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:56:39.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:39.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_28 2024-09-14T05:56:39.062 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_28.status 2024-09-14T05:56:39.062 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_28.status 2024-09-14T05:56:39.062 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:56:39.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_28.status 2024-09-14T05:56:39.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_28.status 2024-09-14T05:56:39.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_28.status 2024-09-14T05:56:39.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:56:39.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_28.status 2024-09-14T05:56:39.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_28 2024-09-14T05:56:39.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:56:39.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:39.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_28 2024-09-14T05:56:39.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_28 2024-09-14T05:56:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 140 2024-09-14T05:56:39.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_28 /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_28.status 2024-09-14T05:56:39.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:56:39.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:39.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_28 2024-09-14T05:56:39.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_28.status 2024-09-14T05:56:39.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_28 --format xml 2024-09-14T05:56:39.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:56:39.965 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:56:39.966 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=140 2024-09-14T05:56:39.966 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:56:39.966 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:56:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:56:40.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_28.status 2024-09-14T05:56:40.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:40.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:40.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_28 2024-09-14T05:56:40.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_28.status 2024-09-14T05:56:40.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-09-14T05:56:40.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:56:40.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:56:40.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:56:40.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:56:40.220 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:56:40.220 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:56:40.220 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:56:40.220 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 140 = '' 2024-09-14T05:56:40.220 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:56:40.220 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:56:40.502 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6913752990902197) > weight (0.25) 2024-09-14T05:56:40.502 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-09-14T05:56:40.503 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-09-14T05:56:40.503 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-09-14T05:56:40.503 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-09-14T05:56:40.503 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T05:56:40.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:2024-09-14T05:56:40.503+0000 7f8affa94640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 51059) UID: 1000 2024-09-14T05:56:40.621 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_28.status 2024-09-14T05:56:40.621 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:40.621 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:40.621 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_28 2024-09-14T05:56:40.621 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_28.status 2024-09-14T05:56:40.621 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-09-14T05:56:40.621 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:56:40.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:56:40.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:56:40.673 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:56:40.673 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:56:40.673 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:56:40.673 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:56:40.673 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 140 = '' 2024-09-14T05:56:40.674 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:56:40.674 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:56:41.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_28.status 2024-09-14T05:56:41.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:41.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:41.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_28 2024-09-14T05:56:41.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_28.status 2024-09-14T05:56:41.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-09-14T05:56:41.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:56:41.523 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:56:41.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=140 2024-09-14T05:56:41.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 140 = 140 2024-09-14T05:56:41.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:56:41.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_28 snap 2024-09-14T05:56:41.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:41.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:41.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_28 2024-09-14T05:56:41.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:56:41.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:56:41.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:56:41.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:56:42.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_28@snap 2024-09-14T05:56:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_28': 2024-09-14T05:56:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:56:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:56:42.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-14T05:56:42.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c18f45aac1 2024-09-14T05:56:42.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c18f45aac1 2024-09-14T05:56:42.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:56:42.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:56:42.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:56:42.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:56:42.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:49:05 2024 2024-09-14T05:56:42.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:49:05 2024 2024-09-14T05:56:42.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:49:05 2024 2024-09-14T05:56:42.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:56:42.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:56:42.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:56:42.577 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 5c71f4af-d4cf-4db0-96fd-a68b7ed67897 2024-09-14T05:56:42.577 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:56:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:56:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_28 snap 2024-09-14T05:56:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_28 2024-09-14T05:56:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:56:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:56:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_28.export 2024-09-14T05:56:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_28.export 2024-09-14T05:56:42.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_28.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_28.export 2024-09-14T05:56:42.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_28@snap /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_28.export 2024-09-14T05:56:42.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:56:42.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_28@snap /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_28.export 2024-09-14T05:56:42.995 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:56:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_28.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_28.export 2024-09-14T05:56:43.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_28.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_28.export 2024-09-14T05:56:43.088 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:56:43.088 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:56:43.088 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_29 2024-09-14T05:56:43.088 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_29 snap 2024-09-14T05:56:43.088 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:56:43.088 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:43.088 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_29 2024-09-14T05:56:43.088 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:56:43.088 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_29@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T05:56:44.926 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_29 2024-09-14T05:56:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_29 2024-09-14T05:56:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_29 started 2024-09-14T05:56:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_29 2024-09-14T05:56:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:56:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:56:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:56:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:56:45.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_29 started 2024-09-14T05:56:45.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:45.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:45.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_29 2024-09-14T05:56:45.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:56:45.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:56:45.928 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:56:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_29 2024-09-14T05:56:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:56:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:56:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:56:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:56:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:56:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:56:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:56:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:56:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:56:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:56:45.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:56:45.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_29 2024-09-14T05:56:45.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:56:45.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:56:45.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:56:45.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:56:45.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:56:45.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:56:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:56:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:56:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:56:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:56:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:56:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:56:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:56:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:56:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:56:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:56:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:56:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:56:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:56:45.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:56:45.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:56:45.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:56:45.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:56:45.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:56:45.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:56:45.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:56:45.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:56:45.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:56:45.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:56:45.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:56:45.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:56:45.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:56:45.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:56:45.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_29 2024-09-14T05:56:46.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:56:46.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:56:46.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:56:46.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:56:46.076 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:56:46.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:56:46.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:56:46.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:56:46.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_29 2024-09-14T05:56:46.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:56:46.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:56:46.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:46.079 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_29 2024-09-14T05:56:46.080 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:56:46.080 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:56:46.080 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_29 2024-09-14T05:56:46.080 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:56:46.080 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:56:46.080 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:46.080 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_29 2024-09-14T05:56:46.080 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_29.status 2024-09-14T05:56:46.080 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_29.status 2024-09-14T05:56:46.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:56:46.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_29.status 2024-09-14T05:56:46.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_29.status 2024-09-14T05:56:46.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_29.status 2024-09-14T05:56:46.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:56:46.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_29.status 2024-09-14T05:56:46.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_29 2024-09-14T05:56:46.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:56:46.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:46.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_29 2024-09-14T05:56:46.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_29 2024-09-14T05:56:46.605 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-09-14T05:56:46.606 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-14T05:56:46.606 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-14T05:56:46.609 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-14T05:56:46.609 INFO:tasks.rbd_mirror_thrash:waiting for 76 secs before thrashing 2024-09-14T05:56:46.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 142 2024-09-14T05:56:46.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_29 /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_29.status 2024-09-14T05:56:46.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:56:46.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:46.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_29 2024-09-14T05:56:46.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_29.status 2024-09-14T05:56:46.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_29 --format xml 2024-09-14T05:56:46.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:56:46.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:56:46.996 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=142 2024-09-14T05:56:46.996 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:56:46.996 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:56:46.996 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:56:47.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_29.status 2024-09-14T05:56:47.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:47.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:47.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_29 2024-09-14T05:56:47.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_29.status 2024-09-14T05:56:47.198 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-09-14T05:56:47.198 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:56:47.247 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:56:47.247 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:56:47.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:56:47.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:56:47.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:56:47.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:56:47.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 142 = '' 2024-09-14T05:56:47.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:56:47.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:56:47.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_29.status 2024-09-14T05:56:47.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:47.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:47.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_29 2024-09-14T05:56:47.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_29.status 2024-09-14T05:56:47.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-09-14T05:56:47.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:56:47.700 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:56:47.700 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:56:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:56:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:56:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:56:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:56:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 142 = '' 2024-09-14T05:56:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:56:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:56:48.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_29.status 2024-09-14T05:56:48.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:48.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:48.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_29 2024-09-14T05:56:48.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_29.status 2024-09-14T05:56:48.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-09-14T05:56:48.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:56:48.552 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:56:48.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=142 2024-09-14T05:56:48.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 142 = 142 2024-09-14T05:56:48.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:56:48.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_29 snap 2024-09-14T05:56:48.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:48.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:48.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_29 2024-09-14T05:56:48.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:56:48.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:56:48.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:56:48.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:56:49.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_29@snap 2024-09-14T05:56:49.604 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_29': 2024-09-14T05:56:49.604 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:56:49.604 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:56:49.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-14T05:56:49.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c1e8e2a2a2 2024-09-14T05:56:49.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c1e8e2a2a2 2024-09-14T05:56:49.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:56:49.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:56:49.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:56:49.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:56:49.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:49:17 2024 2024-09-14T05:56:49.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:49:17 2024 2024-09-14T05:56:49.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:49:17 2024 2024-09-14T05:56:49.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:56:49.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:56:49.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:56:49.606 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 3672d66c-7b41-4af2-9094-129bf558bdc1 2024-09-14T05:56:49.606 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:56:49.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:56:49.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_29 snap 2024-09-14T05:56:49.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:49.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_29 2024-09-14T05:56:49.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:56:49.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:56:49.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_29.export 2024-09-14T05:56:49.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_29.export 2024-09-14T05:56:49.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_29.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_29.export 2024-09-14T05:56:49.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_29@snap /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_29.export 2024-09-14T05:56:49.781 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:56:49.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_29@snap /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_29.export 2024-09-14T05:56:50.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:56:50.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_29.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_29.export 2024-09-14T05:56:50.096 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_29.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_29.export 2024-09-14T05:56:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:56:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:56:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_30 2024-09-14T05:56:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_30 snap 2024-09-14T05:56:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:56:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_30 2024-09-14T05:56:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:56:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_30@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T05:56:51.951 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_30 2024-09-14T05:56:51.951 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:51.951 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:51.951 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_30 2024-09-14T05:56:51.951 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_30 started 2024-09-14T05:56:51.951 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:51.951 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:51.951 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_30 2024-09-14T05:56:51.952 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:56:51.952 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:56:51.952 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:56:51.952 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:56:52.952 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_30 started 2024-09-14T05:56:52.952 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:52.952 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:52.952 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_30 2024-09-14T05:56:52.952 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:56:52.952 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:56:52.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:56:52.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_30 2024-09-14T05:56:52.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:56:52.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:56:52.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:56:52.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:56:52.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:56:52.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:56:52.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:56:52.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:56:52.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:56:52.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:56:52.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:56:52.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_30 2024-09-14T05:56:52.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:56:52.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:56:52.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:56:52.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:56:52.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:56:52.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:56:52.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:56:52.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:56:52.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:56:52.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:56:52.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:56:52.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:56:52.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:56:52.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:56:52.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:56:52.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:56:52.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:56:52.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:56:52.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:56:52.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:56:52.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:56:52.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:56:52.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:56:52.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:56:52.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:56:52.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:56:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:56:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:56:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:56:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:56:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:56:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:56:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:56:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_30 2024-09-14T05:56:53.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:56:53.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:56:53.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:56:53.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:56:53.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:56:53.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:56:53.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:56:53.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:56:53.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_30 2024-09-14T05:56:53.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:56:53.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:56:53.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:53.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_30 2024-09-14T05:56:53.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:56:53.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:56:53.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_30 2024-09-14T05:56:53.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:56:53.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:56:53.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:53.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_30 2024-09-14T05:56:53.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_30.status 2024-09-14T05:56:53.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_30.status 2024-09-14T05:56:53.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:56:53.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_30.status 2024-09-14T05:56:53.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_30.status 2024-09-14T05:56:53.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_30.status 2024-09-14T05:56:53.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:56:53.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_30.status 2024-09-14T05:56:53.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_30 2024-09-14T05:56:53.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:56:53.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:53.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_30 2024-09-14T05:56:53.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_30 2024-09-14T05:56:53.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 144 2024-09-14T05:56:53.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_30 /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_30.status 2024-09-14T05:56:53.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:56:53.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:53.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_30 2024-09-14T05:56:53.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_30.status 2024-09-14T05:56:53.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_30 --format xml 2024-09-14T05:56:53.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:56:54.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:56:54.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=144 2024-09-14T05:56:54.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:56:54.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:56:54.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:56:54.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_30.status 2024-09-14T05:56:54.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:54.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:54.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_30 2024-09-14T05:56:54.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_30.status 2024-09-14T05:56:54.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-09-14T05:56:54.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:56:54.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:56:54.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:56:54.262 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:56:54.262 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:56:54.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:56:54.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:56:54.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 144 = '' 2024-09-14T05:56:54.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:56:54.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:56:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_30.status 2024-09-14T05:56:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_30 2024-09-14T05:56:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_30.status 2024-09-14T05:56:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-09-14T05:56:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:56:54.717 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:56:54.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:56:54.719 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:56:54.719 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:56:54.719 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:56:54.719 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:56:54.719 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 144 = '' 2024-09-14T05:56:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:56:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:56:55.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_30.status 2024-09-14T05:56:55.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:55.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:55.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_30 2024-09-14T05:56:55.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_30.status 2024-09-14T05:56:55.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-09-14T05:56:55.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:56:55.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:56:55.571 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=144 2024-09-14T05:56:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 144 = 144 2024-09-14T05:56:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:56:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_30 snap 2024-09-14T05:56:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_30 2024-09-14T05:56:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:56:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:56:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:56:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:56:56.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_30@snap 2024-09-14T05:56:56.623 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_30': 2024-09-14T05:56:56.623 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:56:56.623 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:56:56.623 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-14T05:56:56.624 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c1359500a9 2024-09-14T05:56:56.624 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c1359500a9 2024-09-14T05:56:56.624 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:56:56.624 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:56:56.624 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:56:56.624 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:56:56.624 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:49:25 2024 2024-09-14T05:56:56.624 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:49:25 2024 2024-09-14T05:56:56.624 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:49:25 2024 2024-09-14T05:56:56.624 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:56:56.624 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:56:56.624 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:56:56.624 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 3f054947-dd92-44d7-bb13-91eab274073e 2024-09-14T05:56:56.625 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:56:56.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:56:56.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_30 snap 2024-09-14T05:56:56.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:56.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_30 2024-09-14T05:56:56.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:56:56.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:56:56.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_30.export 2024-09-14T05:56:56.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_30.export 2024-09-14T05:56:56.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_30.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_30.export 2024-09-14T05:56:56.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_30@snap /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_30.export 2024-09-14T05:56:56.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:56:56.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_30@snap /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_30.export 2024-09-14T05:56:57.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:56:57.035 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_30.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_30.export 2024-09-14T05:56:57.091 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_30.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_30.export 2024-09-14T05:56:57.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:56:57.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:56:57.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_31 2024-09-14T05:56:57.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_31 snap 2024-09-14T05:56:57.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:56:57.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:57.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_31 2024-09-14T05:56:57.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:56:57.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_31@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T05:56:58.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_31 2024-09-14T05:56:58.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:58.958 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:58.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_31 2024-09-14T05:56:58.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_31 started 2024-09-14T05:56:58.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:58.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:58.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_31 2024-09-14T05:56:58.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:56:58.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:56:58.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:56:58.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:56:59.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_31 started 2024-09-14T05:56:59.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:56:59.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:56:59.960 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_31 2024-09-14T05:56:59.960 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:56:59.960 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:56:59.960 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:56:59.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_31 2024-09-14T05:56:59.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:56:59.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:56:59.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:56:59.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:56:59.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:56:59.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:56:59.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:56:59.964 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:56:59.964 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:56:59.966 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:56:59.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:56:59.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_31 2024-09-14T05:56:59.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:56:59.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:56:59.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:56:59.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:56:59.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:56:59.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:56:59.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:56:59.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:56:59.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:56:59.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:56:59.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:56:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:56:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:56:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:56:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:56:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:56:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:56:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:56:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:56:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:56:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:56:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:56:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:56:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:56:59.969 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:56:59.969 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:56:59.969 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:56:59.969 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:56:59.969 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:56:59.969 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:56:59.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:56:59.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:56:59.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:56:59.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_31 2024-09-14T05:57:00.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:57:00.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:57:00.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:57:00.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:57:00.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:57:00.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:57:00.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:57:00.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:57:00.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_31 2024-09-14T05:57:00.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:57:00.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:57:00.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:00.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_31 2024-09-14T05:57:00.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:57:00.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:57:00.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_31 2024-09-14T05:57:00.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:57:00.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:57:00.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:00.109 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_31 2024-09-14T05:57:00.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_31.status 2024-09-14T05:57:00.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_31.status 2024-09-14T05:57:00.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:57:00.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_31.status 2024-09-14T05:57:00.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_31.status 2024-09-14T05:57:00.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_31.status 2024-09-14T05:57:00.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:57:00.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_31.status 2024-09-14T05:57:00.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_31 2024-09-14T05:57:00.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:57:00.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:00.111 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_31 2024-09-14T05:57:00.112 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_31 2024-09-14T05:57:00.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 146 2024-09-14T05:57:00.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_31 /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_31.status 2024-09-14T05:57:00.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:57:00.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:00.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_31 2024-09-14T05:57:00.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_31.status 2024-09-14T05:57:00.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_31 --format xml 2024-09-14T05:57:00.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:57:01.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:57:01.035 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=146 2024-09-14T05:57:01.035 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:57:01.035 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:57:01.035 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:57:01.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_31.status 2024-09-14T05:57:01.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:01.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:01.237 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_31 2024-09-14T05:57:01.237 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_31.status 2024-09-14T05:57:01.237 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-09-14T05:57:01.237 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:57:01.288 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:57:01.288 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:57:01.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:57:01.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:57:01.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:57:01.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:57:01.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 146 = '' 2024-09-14T05:57:01.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:57:01.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:57:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_31.status 2024-09-14T05:57:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_31 2024-09-14T05:57:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_31.status 2024-09-14T05:57:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-09-14T05:57:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:57:01.743 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:57:01.744 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:57:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:57:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:57:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:57:01.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:57:01.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 146 = '' 2024-09-14T05:57:01.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:57:01.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:57:02.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_31.status 2024-09-14T05:57:02.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:02.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:02.547 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_31 2024-09-14T05:57:02.548 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_31.status 2024-09-14T05:57:02.548 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-09-14T05:57:02.548 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:57:02.599 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:57:02.601 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=146 2024-09-14T05:57:02.601 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 146 = 146 2024-09-14T05:57:02.601 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:57:02.601 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_31 snap 2024-09-14T05:57:02.601 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:02.601 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:02.601 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_31 2024-09-14T05:57:02.601 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:57:02.601 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:57:02.601 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:57:02.601 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:57:03.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_31@snap 2024-09-14T05:57:03.658 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_31': 2024-09-14T05:57:03.658 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:57:03.658 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:57:03.658 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-14T05:57:03.658 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c12723ea05 2024-09-14T05:57:03.659 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c12723ea05 2024-09-14T05:57:03.659 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:57:03.659 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:57:03.659 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:57:03.659 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:57:03.659 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:49:37 2024 2024-09-14T05:57:03.659 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:49:37 2024 2024-09-14T05:57:03.659 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:49:37 2024 2024-09-14T05:57:03.659 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:57:03.659 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:57:03.659 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:57:03.659 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 0b236112-fcc0-46d0-8e57-83f45f048a34 2024-09-14T05:57:03.659 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:57:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:57:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_31 snap 2024-09-14T05:57:03.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:03.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_31 2024-09-14T05:57:03.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:57:03.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:57:03.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_31.export 2024-09-14T05:57:03.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_31.export 2024-09-14T05:57:03.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_31.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_31.export 2024-09-14T05:57:03.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_31@snap /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_31.export 2024-09-14T05:57:03.831 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:57:03.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_31@snap /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_31.export 2024-09-14T05:57:04.073 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:57:04.080 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_31.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_31.export 2024-09-14T05:57:04.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_31.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_31.export 2024-09-14T05:57:04.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:57:04.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:57:04.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_32 2024-09-14T05:57:04.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_32 snap 2024-09-14T05:57:04.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:57:04.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:04.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_32 2024-09-14T05:57:04.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:57:04.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_32@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T05:57:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_32 2024-09-14T05:57:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:05.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:05.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_32 2024-09-14T05:57:05.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_32 started 2024-09-14T05:57:05.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:05.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:05.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_32 2024-09-14T05:57:05.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:57:05.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:57:05.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:57:05.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:57:06.513 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-14T05:57:06.514 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-14T05:57:06.514 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-09-14T05:57:06.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-14T05:57:06.514+0000 7f1f1e6bf640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 54798) UID: 1000 2024-09-14T05:57:06.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_32 started 2024-09-14T05:57:06.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:06.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:06.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_32 2024-09-14T05:57:06.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:57:06.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:57:06.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:57:06.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_32 2024-09-14T05:57:06.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:57:06.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:57:06.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:57:06.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:57:06.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:57:06.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:57:06.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:57:06.987 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:57:06.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:57:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:57:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:57:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_32 2024-09-14T05:57:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:57:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:57:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:57:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:57:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:57:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:57:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:57:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:57:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:57:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:57:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:57:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:57:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:57:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:57:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:57:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:57:06.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:57:06.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:57:06.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:57:06.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:57:06.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:57:06.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:57:06.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:57:06.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:57:06.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:57:06.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:57:06.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:57:06.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:57:06.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:57:06.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:57:07.012 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:57:07.012 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:57:07.012 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:57:07.012 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_32 2024-09-14T05:57:07.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:57:07.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:57:07.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:57:07.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:57:07.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:57:07.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:57:07.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:57:07.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:57:07.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_32 2024-09-14T05:57:07.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:57:07.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:57:07.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:07.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_32 2024-09-14T05:57:07.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:57:07.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:57:07.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_32 2024-09-14T05:57:07.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:57:07.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:57:07.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:07.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_32 2024-09-14T05:57:07.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_32.status 2024-09-14T05:57:07.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_32.status 2024-09-14T05:57:07.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:57:07.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_32.status 2024-09-14T05:57:07.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_32.status 2024-09-14T05:57:07.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_32.status 2024-09-14T05:57:07.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:57:07.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_32.status 2024-09-14T05:57:07.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_32 2024-09-14T05:57:07.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:57:07.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:07.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_32 2024-09-14T05:57:07.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_32 2024-09-14T05:57:07.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 148 2024-09-14T05:57:07.996 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_32 /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_32.status 2024-09-14T05:57:07.996 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:57:07.996 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_32 2024-09-14T05:57:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_32.status 2024-09-14T05:57:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_32 --format xml 2024-09-14T05:57:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:57:08.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:57:08.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=148 2024-09-14T05:57:08.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:57:08.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:57:08.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:57:08.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_32.status 2024-09-14T05:57:08.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:08.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:08.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_32 2024-09-14T05:57:08.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_32.status 2024-09-14T05:57:08.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-09-14T05:57:08.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:57:08.301 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:57:08.301 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:57:08.302 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:57:08.302 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:57:08.302 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:57:08.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:57:08.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 148 = '' 2024-09-14T05:57:08.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:57:08.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:57:08.704 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_32.status 2024-09-14T05:57:08.705 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:08.705 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:08.705 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_32 2024-09-14T05:57:08.705 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_32.status 2024-09-14T05:57:08.705 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-09-14T05:57:08.705 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:57:08.756 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:57:08.756 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:57:08.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:57:08.758 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:57:08.758 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:57:08.758 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:57:08.758 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 148 = '' 2024-09-14T05:57:08.758 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:57:08.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:57:09.515 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-14T05:57:09.515 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T05:57:09.515 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-14T05:57:09.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-14T05:57:09.516 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-14T05:57:09.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-14T05:57:09.518 INFO:tasks.rbd_mirror_thrash:waiting for 60 secs before thrashing 2024-09-14T05:57:09.559 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_32.status 2024-09-14T05:57:09.559 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:09.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:09.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_32 2024-09-14T05:57:09.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_32.status 2024-09-14T05:57:09.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-09-14T05:57:09.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:57:09.612 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:57:09.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=148 2024-09-14T05:57:09.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 148 = 148 2024-09-14T05:57:09.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:57:09.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_32 snap 2024-09-14T05:57:09.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:09.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:09.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_32 2024-09-14T05:57:09.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:57:09.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:57:09.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:57:09.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:57:10.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_32@snap 2024-09-14T05:57:10.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_32': 2024-09-14T05:57:10.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:57:10.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:57:10.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-14T05:57:10.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c1b4b6f39d 2024-09-14T05:57:10.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c1b4b6f39d 2024-09-14T05:57:10.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:57:10.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:57:10.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:57:10.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:57:10.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:49:46 2024 2024-09-14T05:57:10.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:49:46 2024 2024-09-14T05:57:10.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:49:46 2024 2024-09-14T05:57:10.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:57:10.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:57:10.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:57:10.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: bcdd7b18-22e7-4ae4-96b5-3c51260f96c7 2024-09-14T05:57:10.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:57:10.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:57:10.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_32 snap 2024-09-14T05:57:10.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:10.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_32 2024-09-14T05:57:10.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:57:10.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:57:10.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_32.export 2024-09-14T05:57:10.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_32.export 2024-09-14T05:57:10.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_32.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_32.export 2024-09-14T05:57:10.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_32@snap /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_32.export 2024-09-14T05:57:10.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:57:10.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_32@snap /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_32.export 2024-09-14T05:57:11.088 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:57:11.095 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_32.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_32.export 2024-09-14T05:57:11.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_32.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_32.export 2024-09-14T05:57:11.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:57:11.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:57:11.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_33 2024-09-14T05:57:11.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_33 snap 2024-09-14T05:57:11.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:57:11.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:11.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_33 2024-09-14T05:57:11.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:57:11.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_33@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T05:57:13.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_33 2024-09-14T05:57:13.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:13.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:13.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_33 2024-09-14T05:57:13.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_33 started 2024-09-14T05:57:13.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:13.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:13.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_33 2024-09-14T05:57:13.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:57:13.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:57:13.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:57:13.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:57:14.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_33 started 2024-09-14T05:57:14.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:14.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:14.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_33 2024-09-14T05:57:14.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:57:14.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:57:14.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:57:14.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_33 2024-09-14T05:57:14.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:57:14.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:57:14.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:57:14.003 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:57:14.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:57:14.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:57:14.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:57:14.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:57:14.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:57:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:57:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:57:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_33 2024-09-14T05:57:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:57:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:57:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:57:14.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:57:14.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:57:14.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:57:14.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:57:14.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:57:14.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:57:14.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:57:14.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:57:14.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:57:14.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:57:14.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:57:14.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:57:14.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:57:14.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:57:14.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:57:14.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:57:14.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:57:14.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:57:14.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:57:14.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:57:14.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:57:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:57:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:57:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:57:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:57:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:57:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:57:14.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:57:14.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:57:14.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:57:14.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_33 2024-09-14T05:57:14.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:57:14.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:57:14.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:57:14.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:57:14.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:57:14.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:57:14.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:57:14.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:57:14.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_33 2024-09-14T05:57:14.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:57:14.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:57:14.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:14.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_33 2024-09-14T05:57:14.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:57:14.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:57:14.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_33 2024-09-14T05:57:14.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:57:14.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:57:14.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:14.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_33 2024-09-14T05:57:14.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_33.status 2024-09-14T05:57:14.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_33.status 2024-09-14T05:57:14.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:57:14.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_33.status 2024-09-14T05:57:14.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_33.status 2024-09-14T05:57:14.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_33.status 2024-09-14T05:57:14.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:57:14.157 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_33.status 2024-09-14T05:57:14.157 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_33 2024-09-14T05:57:14.157 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:57:14.157 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:14.157 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_33 2024-09-14T05:57:14.157 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_33 2024-09-14T05:57:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 150 2024-09-14T05:57:15.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_33 /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_33.status 2024-09-14T05:57:15.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:57:15.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:15.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_33 2024-09-14T05:57:15.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_33.status 2024-09-14T05:57:15.017 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_33 --format xml 2024-09-14T05:57:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:57:15.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:57:15.068 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=150 2024-09-14T05:57:15.068 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:57:15.068 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:57:15.068 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:57:15.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_33.status 2024-09-14T05:57:15.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:15.270 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:15.270 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_33 2024-09-14T05:57:15.270 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_33.status 2024-09-14T05:57:15.270 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2024-09-14T05:57:15.270 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:57:15.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:57:15.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:57:15.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:57:15.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:57:15.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:57:15.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:57:15.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 150 = '' 2024-09-14T05:57:15.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:57:15.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:57:15.722 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_33.status 2024-09-14T05:57:15.722 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:15.722 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:15.722 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_33 2024-09-14T05:57:15.722 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_33.status 2024-09-14T05:57:15.722 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2024-09-14T05:57:15.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:57:15.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:57:15.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:57:15.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:57:15.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:57:15.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:57:15.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:57:15.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 150 = '' 2024-09-14T05:57:15.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:57:15.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:57:16.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_33.status 2024-09-14T05:57:16.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:16.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:16.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_33 2024-09-14T05:57:16.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_33.status 2024-09-14T05:57:16.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2024-09-14T05:57:16.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:57:16.625 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:57:16.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=150 2024-09-14T05:57:16.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 150 = 150 2024-09-14T05:57:16.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:57:16.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_33 snap 2024-09-14T05:57:16.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:16.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:16.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_33 2024-09-14T05:57:16.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:57:16.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:57:16.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:57:16.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:57:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_33@snap 2024-09-14T05:57:17.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_33': 2024-09-14T05:57:17.673 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:57:17.673 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:57:17.673 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-14T05:57:17.673 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c1c1d97f1e 2024-09-14T05:57:17.673 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c1c1d97f1e 2024-09-14T05:57:17.673 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:57:17.673 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:57:17.673 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:57:17.674 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:57:17.674 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:49:53 2024 2024-09-14T05:57:17.674 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:49:53 2024 2024-09-14T05:57:17.674 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:49:53 2024 2024-09-14T05:57:17.674 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:57:17.674 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:57:17.674 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:57:17.674 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: abebcbd1-2502-4a74-8226-d2f7f8e8db34 2024-09-14T05:57:17.674 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:57:17.676 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:57:17.676 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_33 snap 2024-09-14T05:57:17.677 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:17.677 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_33 2024-09-14T05:57:17.677 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:57:17.677 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:57:17.677 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_33.export 2024-09-14T05:57:17.677 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_33.export 2024-09-14T05:57:17.677 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_33.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_33.export 2024-09-14T05:57:17.677 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_33@snap /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_33.export 2024-09-14T05:57:17.857 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:57:17.864 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_33@snap /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_33.export 2024-09-14T05:57:18.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:57:18.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_33.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_33.export 2024-09-14T05:57:18.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_33.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_33.export 2024-09-14T05:57:18.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:57:18.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:57:18.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_34 2024-09-14T05:57:18.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_34 snap 2024-09-14T05:57:18.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:57:18.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:18.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_34 2024-09-14T05:57:18.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:57:18.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_34@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T05:57:20.025 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_34 2024-09-14T05:57:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_34 2024-09-14T05:57:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_34 started 2024-09-14T05:57:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_34 2024-09-14T05:57:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:57:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:57:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:57:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:57:21.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_34 started 2024-09-14T05:57:21.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:21.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:21.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_34 2024-09-14T05:57:21.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:57:21.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:57:21.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:57:21.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_34 2024-09-14T05:57:21.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:57:21.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:57:21.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:57:21.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:57:21.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:57:21.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:57:21.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:57:21.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:57:21.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:57:21.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:57:21.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:57:21.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_34 2024-09-14T05:57:21.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:57:21.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:57:21.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:57:21.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:57:21.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:57:21.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:57:21.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:57:21.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:57:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:57:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:57:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:57:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:57:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:57:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:57:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:57:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:57:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:57:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:57:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:57:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:57:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:57:21.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:57:21.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:57:21.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:57:21.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:57:21.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:57:21.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:57:21.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:57:21.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:57:21.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:57:21.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:57:21.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:57:21.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:57:21.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_34 2024-09-14T05:57:21.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:57:21.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:57:21.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:57:21.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:57:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:57:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:57:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:57:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:57:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_34 2024-09-14T05:57:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:57:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:57:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_34 2024-09-14T05:57:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:57:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:57:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_34 2024-09-14T05:57:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:57:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:57:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_34 2024-09-14T05:57:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_34.status 2024-09-14T05:57:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_34.status 2024-09-14T05:57:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:57:21.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_34.status 2024-09-14T05:57:21.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_34.status 2024-09-14T05:57:21.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_34.status 2024-09-14T05:57:21.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:57:21.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_34.status 2024-09-14T05:57:21.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_34 2024-09-14T05:57:21.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:57:21.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:21.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_34 2024-09-14T05:57:21.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_34 2024-09-14T05:57:22.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 152 2024-09-14T05:57:22.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_34 /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_34.status 2024-09-14T05:57:22.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:57:22.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:22.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_34 2024-09-14T05:57:22.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_34.status 2024-09-14T05:57:22.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_34 --format xml 2024-09-14T05:57:22.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:57:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:57:22.094 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=152 2024-09-14T05:57:22.094 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:57:22.094 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:57:22.094 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:57:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_34.status 2024-09-14T05:57:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:22.296 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_34 2024-09-14T05:57:22.296 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_34.status 2024-09-14T05:57:22.296 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-09-14T05:57:22.296 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:57:22.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:57:22.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=68 2024-09-14T05:57:22.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 152 = 68 2024-09-14T05:57:22.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:57:22.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:57:22.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_34.status 2024-09-14T05:57:22.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:22.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:22.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_34 2024-09-14T05:57:22.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_34.status 2024-09-14T05:57:22.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-09-14T05:57:22.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:57:22.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:57:22.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=68 2024-09-14T05:57:22.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 152 = 68 2024-09-14T05:57:22.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:57:22.807 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:57:23.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_34.status 2024-09-14T05:57:23.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:23.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:23.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_34 2024-09-14T05:57:23.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_34.status 2024-09-14T05:57:23.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-09-14T05:57:23.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:57:23.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:57:23.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:57:23.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:57:23.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:57:23.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:57:23.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:57:23.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 152 = '' 2024-09-14T05:57:23.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:57:23.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-14T05:57:25.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_34.status 2024-09-14T05:57:25.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:25.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:25.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_34 2024-09-14T05:57:25.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_34.status 2024-09-14T05:57:25.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-09-14T05:57:25.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:57:25.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:57:25.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=152 2024-09-14T05:57:25.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 152 = 152 2024-09-14T05:57:25.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:57:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_34 snap 2024-09-14T05:57:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_34 2024-09-14T05:57:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:57:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:57:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:57:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:57:26.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_34@snap 2024-09-14T05:57:26.371 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_34': 2024-09-14T05:57:26.371 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:57:26.371 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:57:26.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-14T05:57:26.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c17d513f7e 2024-09-14T05:57:26.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c17d513f7e 2024-09-14T05:57:26.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:57:26.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:57:26.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:57:26.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:57:26.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:50:07 2024 2024-09-14T05:57:26.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:50:07 2024 2024-09-14T05:57:26.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:50:07 2024 2024-09-14T05:57:26.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:57:26.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:57:26.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:57:26.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 71b41da4-b167-4eb2-a0ea-67c283cfec10 2024-09-14T05:57:26.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:57:26.375 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:57:26.375 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_34 snap 2024-09-14T05:57:26.375 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:26.376 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_34 2024-09-14T05:57:26.376 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:57:26.376 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:57:26.376 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_34.export 2024-09-14T05:57:26.376 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_34.export 2024-09-14T05:57:26.376 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_34.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_34.export 2024-09-14T05:57:26.376 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_34@snap /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_34.export 2024-09-14T05:57:26.546 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:57:26.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_34@snap /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_34.export 2024-09-14T05:57:26.776 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:57:26.782 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_34.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_34.export 2024-09-14T05:57:26.836 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_34.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_34.export 2024-09-14T05:57:26.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:57:26.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:57:26.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_35 2024-09-14T05:57:26.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_35 snap 2024-09-14T05:57:26.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:57:26.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:26.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_35 2024-09-14T05:57:26.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:57:26.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_35@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T05:57:28.041 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_35 2024-09-14T05:57:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_35 2024-09-14T05:57:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_35 started 2024-09-14T05:57:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_35 2024-09-14T05:57:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:57:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:57:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:57:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:57:29.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_35 started 2024-09-14T05:57:29.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:29.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:29.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_35 2024-09-14T05:57:29.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:57:29.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:57:29.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:57:29.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_35 2024-09-14T05:57:29.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:57:29.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:57:29.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:57:29.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:57:29.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:57:29.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:57:29.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:57:29.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:57:29.045 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:57:29.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:57:29.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:57:29.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_35 2024-09-14T05:57:29.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:57:29.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:57:29.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:57:29.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:57:29.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:57:29.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:57:29.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:57:29.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:57:29.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:57:29.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:57:29.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:57:29.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:57:29.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:57:29.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:57:29.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:57:29.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:57:29.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:57:29.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:57:29.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:57:29.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:57:29.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:57:29.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:57:29.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:57:29.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:57:29.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:57:29.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:57:29.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:57:29.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:57:29.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:57:29.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:57:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:57:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:57:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:57:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_35 2024-09-14T05:57:29.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:57:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:57:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:57:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:57:29.193 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:57:29.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:57:29.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:57:29.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:57:29.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_35 2024-09-14T05:57:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:57:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:57:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_35 2024-09-14T05:57:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:57:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:57:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_35 2024-09-14T05:57:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:57:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:57:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_35 2024-09-14T05:57:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_35.status 2024-09-14T05:57:29.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_35.status 2024-09-14T05:57:29.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:57:29.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_35.status 2024-09-14T05:57:29.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_35.status 2024-09-14T05:57:29.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_35.status 2024-09-14T05:57:29.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:57:29.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_35.status 2024-09-14T05:57:29.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_35 2024-09-14T05:57:29.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:57:29.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:29.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_35 2024-09-14T05:57:29.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_35 2024-09-14T05:57:30.041 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 154 2024-09-14T05:57:30.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_35 /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_35.status 2024-09-14T05:57:30.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:57:30.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:30.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_35 2024-09-14T05:57:30.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_35.status 2024-09-14T05:57:30.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_35 --format xml 2024-09-14T05:57:30.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:57:30.102 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:57:30.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=154 2024-09-14T05:57:30.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:57:30.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:57:30.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:57:30.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_35.status 2024-09-14T05:57:30.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:30.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:30.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_35 2024-09-14T05:57:30.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_35.status 2024-09-14T05:57:30.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2024-09-14T05:57:30.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:57:30.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:57:30.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:57:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:57:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:57:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:57:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:57:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 154 = '' 2024-09-14T05:57:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:57:30.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:57:30.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_35.status 2024-09-14T05:57:30.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:30.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:30.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_35 2024-09-14T05:57:30.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_35.status 2024-09-14T05:57:30.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2024-09-14T05:57:30.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:57:30.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:57:30.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:57:30.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:57:30.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:57:30.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:57:30.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:57:30.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 154 = '' 2024-09-14T05:57:30.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:57:30.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:57:31.621 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_35.status 2024-09-14T05:57:31.621 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:31.621 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:31.621 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_35 2024-09-14T05:57:31.621 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_35.status 2024-09-14T05:57:31.621 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2024-09-14T05:57:31.621 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:57:31.667 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:57:31.669 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=154 2024-09-14T05:57:31.669 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 154 = 154 2024-09-14T05:57:31.669 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:57:31.669 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_35 snap 2024-09-14T05:57:31.669 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:31.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:31.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_35 2024-09-14T05:57:31.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:57:31.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:57:31.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:57:31.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:57:32.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_35@snap 2024-09-14T05:57:32.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_35': 2024-09-14T05:57:32.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:57:32.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:57:32.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-14T05:57:32.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c1a4714718 2024-09-14T05:57:32.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c1a4714718 2024-09-14T05:57:32.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:57:32.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:57:32.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:57:32.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:57:32.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:50:22 2024 2024-09-14T05:57:32.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:50:22 2024 2024-09-14T05:57:32.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:50:22 2024 2024-09-14T05:57:32.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:57:32.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:57:32.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:57:32.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 6249a2fd-1afb-4dee-89ca-39ba30cbe74d 2024-09-14T05:57:32.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:57:32.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:57:32.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_35 snap 2024-09-14T05:57:32.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:32.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_35 2024-09-14T05:57:32.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:57:32.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:57:32.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_35.export 2024-09-14T05:57:32.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_35.export 2024-09-14T05:57:32.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_35.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_35.export 2024-09-14T05:57:32.719 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_35@snap /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_35.export 2024-09-14T05:57:32.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:57:32.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_35@snap /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_35.export 2024-09-14T05:57:33.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:57:33.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_35.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_35.export 2024-09-14T05:57:33.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_35.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_35.export 2024-09-14T05:57:33.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:57:33.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:57:33.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_36 2024-09-14T05:57:33.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_36 snap 2024-09-14T05:57:33.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:57:33.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:33.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_36 2024-09-14T05:57:33.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:57:33.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_36@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T05:57:35.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_36 2024-09-14T05:57:35.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:35.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:35.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_36 2024-09-14T05:57:35.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_36 started 2024-09-14T05:57:35.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:35.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:35.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_36 2024-09-14T05:57:35.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:57:35.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:57:35.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:57:35.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:57:36.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_36 started 2024-09-14T05:57:36.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:36.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:36.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_36 2024-09-14T05:57:36.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:57:36.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:57:36.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:57:36.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_36 2024-09-14T05:57:36.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:57:36.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:57:36.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:57:36.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:57:36.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:57:36.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:57:36.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:57:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:57:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:57:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:57:36.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:57:36.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_36 2024-09-14T05:57:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:57:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:57:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:57:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:57:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:57:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:57:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:57:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:57:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:57:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:57:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:57:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:57:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:57:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:57:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:57:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:57:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:57:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:57:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:57:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:57:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:57:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:57:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:57:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:57:36.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:57:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:57:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:57:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:57:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:57:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:57:36.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:57:36.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:57:36.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:57:36.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_36 2024-09-14T05:57:36.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:57:36.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:57:36.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:57:36.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:57:36.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:57:36.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:57:36.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:57:36.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:57:36.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_36 2024-09-14T05:57:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:57:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:57:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_36 2024-09-14T05:57:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:57:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:57:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_36 2024-09-14T05:57:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:57:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:57:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_36 2024-09-14T05:57:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_36.status 2024-09-14T05:57:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_36.status 2024-09-14T05:57:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:57:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_36.status 2024-09-14T05:57:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_36.status 2024-09-14T05:57:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_36.status 2024-09-14T05:57:36.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:57:36.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_36.status 2024-09-14T05:57:36.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_36 2024-09-14T05:57:36.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:57:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_36 2024-09-14T05:57:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_36 2024-09-14T05:57:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 156 2024-09-14T05:57:37.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_36 /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_36.status 2024-09-14T05:57:37.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:57:37.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:37.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_36 2024-09-14T05:57:37.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_36.status 2024-09-14T05:57:37.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_36 --format xml 2024-09-14T05:57:37.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:57:37.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:57:37.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=156 2024-09-14T05:57:37.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:57:37.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:57:37.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:57:37.327 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_36.status 2024-09-14T05:57:37.328 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:37.328 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:37.328 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_36 2024-09-14T05:57:37.328 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_36.status 2024-09-14T05:57:37.328 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2024-09-14T05:57:37.328 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:57:37.380 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:57:37.381 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=70 2024-09-14T05:57:37.381 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 156 = 70 2024-09-14T05:57:37.382 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:57:37.382 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:57:37.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_36.status 2024-09-14T05:57:37.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:37.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:37.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_36 2024-09-14T05:57:37.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_36.status 2024-09-14T05:57:37.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2024-09-14T05:57:37.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:57:37.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:57:37.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=70 2024-09-14T05:57:37.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 156 = 70 2024-09-14T05:57:37.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:57:37.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:57:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_36.status 2024-09-14T05:57:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_36 2024-09-14T05:57:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_36.status 2024-09-14T05:57:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2024-09-14T05:57:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:57:38.698 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:57:38.699 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:57:38.700 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:57:38.700 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:57:38.700 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:57:38.701 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:57:38.701 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 156 = '' 2024-09-14T05:57:38.701 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:57:38.701 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-14T05:57:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_36.status 2024-09-14T05:57:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_36 2024-09-14T05:57:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_36.status 2024-09-14T05:57:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2024-09-14T05:57:40.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:57:40.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:57:40.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=156 2024-09-14T05:57:40.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 156 = 156 2024-09-14T05:57:40.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:57:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_36 snap 2024-09-14T05:57:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_36 2024-09-14T05:57:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:57:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:57:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:57:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:57:41.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_36@snap 2024-09-14T05:57:41.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_36': 2024-09-14T05:57:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:57:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:57:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-14T05:57:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c1716916d2 2024-09-14T05:57:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c1716916d2 2024-09-14T05:57:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:57:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:57:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:57:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:57:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:50:33 2024 2024-09-14T05:57:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:50:33 2024 2024-09-14T05:57:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:50:33 2024 2024-09-14T05:57:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:57:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:57:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:57:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 702a468e-cfaa-4567-a91a-60fa43b31b1f 2024-09-14T05:57:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:57:41.408 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:57:41.408 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_36 snap 2024-09-14T05:57:41.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:41.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_36 2024-09-14T05:57:41.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:57:41.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:57:41.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_36.export 2024-09-14T05:57:41.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_36.export 2024-09-14T05:57:41.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_36.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_36.export 2024-09-14T05:57:41.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_36@snap /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_36.export 2024-09-14T05:57:41.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:57:41.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_36@snap /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_36.export 2024-09-14T05:57:41.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:57:41.815 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_36.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_36.export 2024-09-14T05:57:41.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_36.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_36.export 2024-09-14T05:57:41.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:57:41.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:57:41.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_37 2024-09-14T05:57:41.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_37 snap 2024-09-14T05:57:41.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:57:41.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:41.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_37 2024-09-14T05:57:41.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:57:41.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_37@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T05:57:43.080 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_37 2024-09-14T05:57:43.080 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_37 2024-09-14T05:57:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_37 started 2024-09-14T05:57:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_37 2024-09-14T05:57:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:57:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:57:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:57:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:57:44.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_37 started 2024-09-14T05:57:44.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:44.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:44.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_37 2024-09-14T05:57:44.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:57:44.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:57:44.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:57:44.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_37 2024-09-14T05:57:44.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:57:44.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:57:44.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:57:44.084 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:57:44.084 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:57:44.084 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:57:44.084 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:57:44.084 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:57:44.084 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:57:44.084 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:57:44.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:57:44.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_37 2024-09-14T05:57:44.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:57:44.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:57:44.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:57:44.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:57:44.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:57:44.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:57:44.088 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:57:44.088 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:57:44.088 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:57:44.088 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:57:44.088 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:57:44.088 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:57:44.088 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:57:44.088 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:57:44.088 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:57:44.088 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:57:44.088 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:57:44.088 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:57:44.088 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:57:44.089 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:57:44.089 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:57:44.089 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:57:44.089 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:57:44.089 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:57:44.089 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:57:44.089 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:57:44.089 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:57:44.089 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:57:44.089 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:57:44.089 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:57:44.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:57:44.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:57:44.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:57:44.108 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_37 2024-09-14T05:57:44.229 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:57:44.229 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:57:44.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:57:44.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:57:44.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:57:44.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:57:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:57:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:57:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_37 2024-09-14T05:57:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:57:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:57:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_37 2024-09-14T05:57:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:57:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:57:44.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_37 2024-09-14T05:57:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:57:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:57:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_37 2024-09-14T05:57:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_37.status 2024-09-14T05:57:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_37.status 2024-09-14T05:57:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:57:44.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_37.status 2024-09-14T05:57:44.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_37.status 2024-09-14T05:57:44.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_37.status 2024-09-14T05:57:44.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:57:44.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_37.status 2024-09-14T05:57:44.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_37 2024-09-14T05:57:44.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:57:44.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:44.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_37 2024-09-14T05:57:44.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_37 2024-09-14T05:57:45.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 158 2024-09-14T05:57:45.097 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_37 /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_37.status 2024-09-14T05:57:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:57:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_37 2024-09-14T05:57:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_37.status 2024-09-14T05:57:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_37 --format xml 2024-09-14T05:57:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:57:45.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:57:45.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=158 2024-09-14T05:57:45.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:57:45.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:57:45.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:57:45.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_37.status 2024-09-14T05:57:45.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:45.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:45.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_37 2024-09-14T05:57:45.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_37.status 2024-09-14T05:57:45.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2024-09-14T05:57:45.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:57:45.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:57:45.399 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=71 2024-09-14T05:57:45.399 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 158 = 71 2024-09-14T05:57:45.399 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:57:45.399 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:57:45.801 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_37.status 2024-09-14T05:57:45.801 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:45.801 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:45.801 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_37 2024-09-14T05:57:45.801 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_37.status 2024-09-14T05:57:45.801 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2024-09-14T05:57:45.801 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:57:45.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:57:45.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=71 2024-09-14T05:57:45.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 158 = 71 2024-09-14T05:57:45.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:57:45.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:57:46.652 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_37.status 2024-09-14T05:57:46.652 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:46.652 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:46.652 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_37 2024-09-14T05:57:46.652 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_37.status 2024-09-14T05:57:46.652 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2024-09-14T05:57:46.652 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:57:46.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:57:46.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:57:46.708 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:57:46.708 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:57:46.708 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:57:46.708 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:57:46.709 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 158 = '' 2024-09-14T05:57:46.709 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:57:46.709 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-14T05:57:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_37.status 2024-09-14T05:57:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_37 2024-09-14T05:57:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_37.status 2024-09-14T05:57:48.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2024-09-14T05:57:48.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:57:48.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:57:48.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=158 2024-09-14T05:57:48.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 158 = 158 2024-09-14T05:57:48.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:57:48.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_37 snap 2024-09-14T05:57:48.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:48.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:48.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_37 2024-09-14T05:57:48.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:57:48.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:57:48.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:57:48.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:57:49.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_37@snap 2024-09-14T05:57:49.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_37': 2024-09-14T05:57:49.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:57:49.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:57:49.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-14T05:57:49.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c19d713204 2024-09-14T05:57:49.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c19d713204 2024-09-14T05:57:49.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:57:49.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:57:49.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:57:49.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:57:49.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:50:39 2024 2024-09-14T05:57:49.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:50:39 2024 2024-09-14T05:57:49.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:50:39 2024 2024-09-14T05:57:49.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:57:49.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:57:49.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:57:49.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 07ff7849-860f-4f87-8bed-88f824442785 2024-09-14T05:57:49.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:57:49.423 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:57:49.423 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_37 snap 2024-09-14T05:57:49.423 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:49.423 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_37 2024-09-14T05:57:49.423 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:57:49.424 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:57:49.424 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_37.export 2024-09-14T05:57:49.424 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_37.export 2024-09-14T05:57:49.424 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_37.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_37.export 2024-09-14T05:57:49.424 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_37@snap /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_37.export 2024-09-14T05:57:49.594 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:57:49.601 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_37@snap /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_37.export 2024-09-14T05:57:49.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:57:49.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_37.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_37.export 2024-09-14T05:57:49.900 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_37.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_37.export 2024-09-14T05:57:49.931 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:57:49.931 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:57:49.931 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_38 2024-09-14T05:57:49.931 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_38 snap 2024-09-14T05:57:49.931 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:57:49.932 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:49.932 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_38 2024-09-14T05:57:49.932 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:57:49.932 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_38@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T05:57:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_38 2024-09-14T05:57:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_38 2024-09-14T05:57:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_38 started 2024-09-14T05:57:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_38 2024-09-14T05:57:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:57:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:57:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:57:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:57:52.100 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_38 started 2024-09-14T05:57:52.100 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:52.100 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:52.100 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_38 2024-09-14T05:57:52.100 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:57:52.100 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:57:52.101 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:57:52.102 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_38 2024-09-14T05:57:52.102 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:57:52.102 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:57:52.102 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:57:52.102 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:57:52.102 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:57:52.102 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:57:52.102 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:57:52.102 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:57:52.102 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:57:52.102 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:57:52.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:57:52.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_38 2024-09-14T05:57:52.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:57:52.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:57:52.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:57:52.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:57:52.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:57:52.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:57:52.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:57:52.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:57:52.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:57:52.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:57:52.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:57:52.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:57:52.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:57:52.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:57:52.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:57:52.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:57:52.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:57:52.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:57:52.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:57:52.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:57:52.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:57:52.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:57:52.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:57:52.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:57:52.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:57:52.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:57:52.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:57:52.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:57:52.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:57:52.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:57:52.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:57:52.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:57:52.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:57:52.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_38 2024-09-14T05:57:52.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:57:52.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:57:52.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:57:52.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:57:52.250 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:57:52.251 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:57:52.251 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:57:52.251 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:57:52.251 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_38 2024-09-14T05:57:52.251 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:57:52.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:57:52.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:52.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_38 2024-09-14T05:57:52.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:57:52.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:57:52.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_38 2024-09-14T05:57:52.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:57:52.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:57:52.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_38 2024-09-14T05:57:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_38.status 2024-09-14T05:57:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_38.status 2024-09-14T05:57:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:57:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_38.status 2024-09-14T05:57:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_38.status 2024-09-14T05:57:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_38.status 2024-09-14T05:57:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:57:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_38.status 2024-09-14T05:57:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_38 2024-09-14T05:57:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:57:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_38 2024-09-14T05:57:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_38 2024-09-14T05:57:53.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 160 2024-09-14T05:57:53.117 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_38 /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_38.status 2024-09-14T05:57:53.117 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:57:53.117 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:53.117 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_38 2024-09-14T05:57:53.117 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_38.status 2024-09-14T05:57:53.117 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_38 --format xml 2024-09-14T05:57:53.117 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:57:53.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:57:53.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=160 2024-09-14T05:57:53.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:57:53.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:57:53.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:57:53.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_38.status 2024-09-14T05:57:53.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:53.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:53.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_38 2024-09-14T05:57:53.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_38.status 2024-09-14T05:57:53.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2024-09-14T05:57:53.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:57:53.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:57:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=72 2024-09-14T05:57:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 160 = 72 2024-09-14T05:57:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:57:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:57:53.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_38.status 2024-09-14T05:57:53.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:53.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:53.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_38 2024-09-14T05:57:53.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_38.status 2024-09-14T05:57:53.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2024-09-14T05:57:53.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:57:53.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:57:53.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=72 2024-09-14T05:57:53.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 160 = 72 2024-09-14T05:57:53.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:57:53.869 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:57:54.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_38.status 2024-09-14T05:57:54.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:54.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:54.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_38 2024-09-14T05:57:54.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_38.status 2024-09-14T05:57:54.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2024-09-14T05:57:54.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:57:54.724 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:57:54.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:57:54.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:57:54.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:57:54.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:57:54.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:57:54.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 160 = '' 2024-09-14T05:57:54.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:57:54.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-14T05:57:56.328 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_38.status 2024-09-14T05:57:56.329 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:56.329 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:56.329 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_38 2024-09-14T05:57:56.329 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_38.status 2024-09-14T05:57:56.329 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2024-09-14T05:57:56.329 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:57:56.383 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:57:56.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=160 2024-09-14T05:57:56.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 160 = 160 2024-09-14T05:57:56.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:57:56.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_38 snap 2024-09-14T05:57:56.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:56.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:56.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_38 2024-09-14T05:57:56.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:57:56.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:57:56.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:57:56.386 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:57:57.386 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_38@snap 2024-09-14T05:57:57.435 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_38': 2024-09-14T05:57:57.436 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:57:57.436 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:57:57.436 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-14T05:57:57.436 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c1cc82d3db 2024-09-14T05:57:57.436 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c1cc82d3db 2024-09-14T05:57:57.436 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:57:57.436 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:57:57.436 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:57:57.436 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:57:57.436 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:50:45 2024 2024-09-14T05:57:57.436 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:50:45 2024 2024-09-14T05:57:57.436 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:50:45 2024 2024-09-14T05:57:57.436 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:57:57.437 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:57:57.437 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:57:57.437 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 4cab209c-76f4-435e-9f27-921251cc9286 2024-09-14T05:57:57.437 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:57:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:57:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_38 snap 2024-09-14T05:57:57.440 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:57.440 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_38 2024-09-14T05:57:57.440 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:57:57.440 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:57:57.440 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_38.export 2024-09-14T05:57:57.440 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_38.export 2024-09-14T05:57:57.440 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_38.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_38.export 2024-09-14T05:57:57.440 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_38@snap /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_38.export 2024-09-14T05:57:57.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:57:57.611 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_38@snap /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_38.export 2024-09-14T05:57:57.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:57:57.851 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_38.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_38.export 2024-09-14T05:57:57.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_38.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_38.export 2024-09-14T05:57:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:57:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:57:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_39 2024-09-14T05:57:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_39 snap 2024-09-14T05:57:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:57:57.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:57.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_39 2024-09-14T05:57:57.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:57:57.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_39@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T05:57:59.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_39 2024-09-14T05:57:59.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:59.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:59.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_39 2024-09-14T05:57:59.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_39 started 2024-09-14T05:57:59.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:57:59.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:57:59.135 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_39 2024-09-14T05:57:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:57:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:57:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:57:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:58:00.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_39 started 2024-09-14T05:58:00.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:58:00.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:00.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_39 2024-09-14T05:58:00.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:58:00.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:58:00.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:58:00.137 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_39 2024-09-14T05:58:00.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:58:00.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:58:00.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:58:00.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:58:00.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:58:00.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:58:00.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:58:00.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:58:00.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:58:00.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:58:00.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:58:00.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_39 2024-09-14T05:58:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:58:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:58:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:58:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:58:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:58:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:58:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:58:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:58:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:58:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:58:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:58:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:58:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:58:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:58:00.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:58:00.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:58:00.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:58:00.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:58:00.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:58:00.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:58:00.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:58:00.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:58:00.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:58:00.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:58:00.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:58:00.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:58:00.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:58:00.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:58:00.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:58:00.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:58:00.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:58:00.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:58:00.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:58:00.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_39 2024-09-14T05:58:00.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:58:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:58:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:58:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:58:00.287 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:58:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:58:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:58:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:58:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_39 2024-09-14T05:58:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:58:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:58:00.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:00.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_39 2024-09-14T05:58:00.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:58:00.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:58:00.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_39 2024-09-14T05:58:00.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:58:00.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:58:00.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:00.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_39 2024-09-14T05:58:00.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_39.status 2024-09-14T05:58:00.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_39.status 2024-09-14T05:58:00.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:58:00.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_39.status 2024-09-14T05:58:00.291 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_39.status 2024-09-14T05:58:00.291 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_39.status 2024-09-14T05:58:00.291 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:58:00.291 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_39.status 2024-09-14T05:58:00.291 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_39 2024-09-14T05:58:00.291 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:58:00.291 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:00.291 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_39 2024-09-14T05:58:00.291 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_39 2024-09-14T05:58:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 162 2024-09-14T05:58:01.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_39 /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_39.status 2024-09-14T05:58:01.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:58:01.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:01.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_39 2024-09-14T05:58:01.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_39.status 2024-09-14T05:58:01.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_39 --format xml 2024-09-14T05:58:01.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:58:01.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:58:01.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=162 2024-09-14T05:58:01.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:58:01.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:58:01.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:58:01.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_39.status 2024-09-14T05:58:01.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:58:01.398 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:01.398 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_39 2024-09-14T05:58:01.398 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_39.status 2024-09-14T05:58:01.398 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2024-09-14T05:58:01.398 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:58:01.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:58:01.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=73 2024-09-14T05:58:01.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 162 = 73 2024-09-14T05:58:01.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:58:01.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:58:01.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_39.status 2024-09-14T05:58:01.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:58:01.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:01.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_39 2024-09-14T05:58:01.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_39.status 2024-09-14T05:58:01.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2024-09-14T05:58:01.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:58:01.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:58:01.899 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=73 2024-09-14T05:58:01.899 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 162 = 73 2024-09-14T05:58:01.899 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:58:01.899 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:58:02.610 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-14T05:58:02.610 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-14T05:58:02.611 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-09-14T05:58:02.611 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-14T05:58:02.611 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T05:58:02.612 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-14T05:58:02.611+0000 7f5848ae0640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 42970) UID: 1000 2024-09-14T05:58:02.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_39.status 2024-09-14T05:58:02.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:58:02.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:02.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_39 2024-09-14T05:58:02.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_39.status 2024-09-14T05:58:02.704 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:58:02.704 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2024-09-14T05:58:02.761 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:58:02.761 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:58:02.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:58:02.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:58:02.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:58:02.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:58:02.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 162 = '' 2024-09-14T05:58:02.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:58:02.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-14T05:58:04.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_39.status 2024-09-14T05:58:04.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:58:04.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:04.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_39 2024-09-14T05:58:04.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_39.status 2024-09-14T05:58:04.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2024-09-14T05:58:04.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:58:04.425 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:58:04.425 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:58:04.426 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:58:04.426 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:58:04.426 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:58:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:58:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 162 = '' 2024-09-14T05:58:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:58:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 2 2024-09-14T05:58:06.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_39.status 2024-09-14T05:58:06.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:58:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_39 2024-09-14T05:58:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_39.status 2024-09-14T05:58:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2024-09-14T05:58:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:58:06.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:58:06.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:58:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:58:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:58:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:58:06.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:58:06.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 162 = '' 2024-09-14T05:58:06.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:58:06.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 2 2024-09-14T05:58:08.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_39.status 2024-09-14T05:58:08.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:58:08.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:08.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_39 2024-09-14T05:58:08.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_39.status 2024-09-14T05:58:08.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2024-09-14T05:58:08.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:58:08.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:58:08.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:58:08.542 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:58:08.542 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:58:08.542 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:58:08.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:58:08.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 162 = '' 2024-09-14T05:58:08.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:58:08.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 4 2024-09-14T05:58:08.712 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-14T05:58:08.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-14T05:58:08.713 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-14T05:58:08.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-14T05:58:08.716 INFO:tasks.rbd_mirror_thrash:waiting for 115 secs before thrashing 2024-09-14T05:58:09.519 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6251506806492796) > weight (0.25) 2024-09-14T05:58:09.520 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8289077249861413) > weight (0.25) 2024-09-14T05:58:09.520 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-09-14T05:58:09.520 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-09-14T05:58:09.520 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-09-14T05:58:09.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-14T05:58:09.520+0000 7fae356b4640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 49683) UID: 1000 2024-09-14T05:58:12.520 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-09-14T05:58:12.521 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T05:58:12.521 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-09-14T05:58:12.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-14T05:58:12.521 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-14T05:58:12.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-14T05:58:12.524 INFO:tasks.rbd_mirror_thrash:waiting for 108 secs before thrashing 2024-09-14T05:58:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_39.status 2024-09-14T05:58:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:58:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_39 2024-09-14T05:58:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_39.status 2024-09-14T05:58:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:58:12.545 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2024-09-14T05:58:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:58:12.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=162 2024-09-14T05:58:12.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 162 = 162 2024-09-14T05:58:12.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:58:12.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_39 snap 2024-09-14T05:58:12.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:58:12.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:12.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_39 2024-09-14T05:58:12.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:58:12.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:58:12.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:58:12.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:58:13.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_39@snap 2024-09-14T05:58:13.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_39': 2024-09-14T05:58:13.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:58:13.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:58:13.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-14T05:58:13.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c117f591a0 2024-09-14T05:58:13.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c117f591a0 2024-09-14T05:58:13.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:58:13.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:58:13.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:58:13.651 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:58:13.652 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:50:53 2024 2024-09-14T05:58:13.652 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:50:53 2024 2024-09-14T05:58:13.652 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:50:53 2024 2024-09-14T05:58:13.652 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:58:13.652 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:58:13.652 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:58:13.652 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 37b3b339-d8a4-4dd0-ac91-56b0d1df8ebd 2024-09-14T05:58:13.652 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:58:13.655 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:58:13.655 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_39 snap 2024-09-14T05:58:13.655 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:13.655 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_39 2024-09-14T05:58:13.655 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:58:13.655 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:58:13.655 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_39.export 2024-09-14T05:58:13.655 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_39.export 2024-09-14T05:58:13.655 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_39.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_39.export 2024-09-14T05:58:13.656 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_39@snap /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_39.export 2024-09-14T05:58:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:58:13.830 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_39@snap /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_39.export 2024-09-14T05:58:14.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:58:14.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_39.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_39.export 2024-09-14T05:58:14.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_39.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_39.export 2024-09-14T05:58:14.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:58:14.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:58:14.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_40 2024-09-14T05:58:14.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_40 snap 2024-09-14T05:58:14.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:58:14.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:14.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_40 2024-09-14T05:58:14.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:58:14.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_40@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T05:58:16.158 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_40 2024-09-14T05:58:16.158 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:58:16.158 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:16.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_40 2024-09-14T05:58:16.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_40 started 2024-09-14T05:58:16.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:58:16.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:16.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_40 2024-09-14T05:58:16.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:58:16.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:58:16.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:58:16.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:58:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_40 started 2024-09-14T05:58:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:58:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:17.160 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_40 2024-09-14T05:58:17.160 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:58:17.160 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:58:17.160 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:58:17.161 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_40 2024-09-14T05:58:17.161 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:58:17.161 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:58:17.161 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:58:17.161 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:58:17.161 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:58:17.161 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:58:17.161 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:58:17.161 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:58:17.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:58:17.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:58:17.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:58:17.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_40 2024-09-14T05:58:17.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:58:17.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:58:17.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:58:17.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:58:17.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:58:17.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:58:17.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:58:17.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:58:17.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:58:17.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:58:17.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:58:17.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:58:17.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:58:17.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:58:17.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:58:17.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:58:17.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:58:17.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:58:17.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:58:17.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:58:17.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:58:17.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:58:17.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:58:17.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:58:17.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:58:17.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:58:17.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:58:17.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:58:17.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:58:17.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:58:17.186 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:58:17.186 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:58:17.186 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:58:17.187 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_40 2024-09-14T05:58:17.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2024-09-14T05:58:17.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:58:17.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:58:17.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:58:17.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_40 2024-09-14T05:58:17.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-14T05:58:17.270 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:58:17.270 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:58:17.270 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T05:58:17.270 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-14T05:58:17.270 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:58:17.270 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:58:17.270 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:58:17.270 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-14T05:58:17.270 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T05:58:17.270 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:58:17.270 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:58:17.270 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-14T05:58:17.270 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:58:17.270 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T05:58:17.271 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-14T05:58:17.271 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:58:17.271 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:58:17.271 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T05:58:17.271 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-14T05:58:17.271 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:58:17.271 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:58:17.271 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:58:17.271 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:58:17.271 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T05:58:17.271 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:58:17.271 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:58:17.271 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-14T05:58:17.272 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-14T05:58:17.272 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T05:58:17.288 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:58:17.288 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:58:17.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:58:17.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_40 2024-09-14T05:58:17.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2024-09-14T05:58:17.363 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:58:17.363 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:58:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:58:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_40 2024-09-14T05:58:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-14T05:58:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:58:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:58:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T05:58:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-14T05:58:17.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:58:17.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:58:17.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:58:17.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-14T05:58:17.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T05:58:17.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:58:17.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:58:17.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-14T05:58:17.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:58:17.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T05:58:17.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-14T05:58:17.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:58:17.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:58:17.368 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T05:58:17.368 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-14T05:58:17.368 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:58:17.368 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:58:17.368 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:58:17.368 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:58:17.368 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T05:58:17.368 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:58:17.368 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:58:17.368 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-14T05:58:17.368 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-14T05:58:17.368 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T05:58:17.384 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:58:17.384 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:58:17.384 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:58:17.384 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_40 2024-09-14T05:58:17.456 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2024-09-14T05:58:17.456 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:58:17.456 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:58:17.460 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:58:17.460 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_40 2024-09-14T05:58:17.460 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:3 2024-09-14T05:58:17.460 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:58:17.460 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:58:17.460 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-14T05:58:17.460 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:3 2024-09-14T05:58:17.460 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:58:17.460 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:58:17.460 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:58:17.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=3 2024-09-14T05:58:17.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-14T05:58:17.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:58:17.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:58:17.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=3 2024-09-14T05:58:17.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-14T05:58:17.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-14T05:58:17.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:3 2024-09-14T05:58:17.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:58:17.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:58:17.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-14T05:58:17.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:3 2024-09-14T05:58:17.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:58:17.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:58:17.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:58:17.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-14T05:58:17.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-14T05:58:17.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:58:17.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:58:17.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=3 2024-09-14T05:58:17.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=3 2024-09-14T05:58:17.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-14T05:58:17.478 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T05:58:17.478 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T05:58:17.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T05:58:17.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_40 2024-09-14T05:58:17.594 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:58:17.594 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:58:17.594 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:58:17.594 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:58:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:58:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:58:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:58:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:58:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_40 2024-09-14T05:58:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:58:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:58:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_40 2024-09-14T05:58:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:58:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:58:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_40 2024-09-14T05:58:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:58:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:58:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_40 2024-09-14T05:58:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_40.status 2024-09-14T05:58:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_40.status 2024-09-14T05:58:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:58:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_40.status 2024-09-14T05:58:17.599 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_40.status 2024-09-14T05:58:17.599 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_40.status 2024-09-14T05:58:17.600 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:58:17.600 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_40.status 2024-09-14T05:58:17.600 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_40 2024-09-14T05:58:17.600 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:58:17.600 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:17.600 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_40 2024-09-14T05:58:17.600 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_40 2024-09-14T05:58:18.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 164 2024-09-14T05:58:18.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_40 /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_40.status 2024-09-14T05:58:18.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:58:18.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:18.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_40 2024-09-14T05:58:18.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_40.status 2024-09-14T05:58:18.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_40 --format xml 2024-09-14T05:58:18.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:58:18.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:58:18.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=164 2024-09-14T05:58:18.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:58:18.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:58:18.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:58:18.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_40.status 2024-09-14T05:58:18.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:58:18.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:18.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_40 2024-09-14T05:58:18.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_40.status 2024-09-14T05:58:18.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-09-14T05:58:18.416 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:58:18.464 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:58:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=74 2024-09-14T05:58:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 164 = 74 2024-09-14T05:58:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:58:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:58:18.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_40.status 2024-09-14T05:58:18.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:58:18.867 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:18.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_40 2024-09-14T05:58:18.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_40.status 2024-09-14T05:58:18.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-09-14T05:58:18.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:58:18.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:58:18.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=74 2024-09-14T05:58:18.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 164 = 74 2024-09-14T05:58:18.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:58:18.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:58:19.719 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_40.status 2024-09-14T05:58:19.719 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:58:19.719 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_40 2024-09-14T05:58:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_40.status 2024-09-14T05:58:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-09-14T05:58:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:58:19.773 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:58:19.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:58:19.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:58:19.776 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:58:19.776 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:58:19.776 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:58:19.776 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 164 = '' 2024-09-14T05:58:19.776 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:58:19.776 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-14T05:58:20.999 INFO:tasks.ceph.cluster1.osd.2.smithi143.stderr:2024-09-14T05:58:20.997+0000 7ff5a47b4640 -1 reset not still connected to 0x55c40c1389c0 2024-09-14T05:58:20.999 INFO:tasks.ceph.cluster1.osd.2.smithi143.stderr:2024-09-14T05:58:20.997+0000 7ff5a47b4640 -1 reset not still connected to 0x55c40c9a96c0 2024-09-14T05:58:21.001 INFO:tasks.ceph.cluster1.osd.2.smithi143.stderr:2024-09-14T05:58:21.000+0000 7ff5a47b4640 -1 reset not still connected to 0x55c40d3ba340 2024-09-14T05:58:21.001 INFO:tasks.ceph.cluster1.osd.2.smithi143.stderr:2024-09-14T05:58:21.000+0000 7ff5a47b4640 -1 reset not still connected to 0x55c40d3ba4e0 2024-09-14T05:58:21.001 INFO:tasks.ceph.cluster1.osd.2.smithi143.stderr:2024-09-14T05:58:21.000+0000 7ff5a47b4640 -1 reset not still connected to 0x55c40d3ba5b0 2024-09-14T05:58:21.001 INFO:tasks.ceph.cluster1.osd.2.smithi143.stderr:2024-09-14T05:58:21.000+0000 7ff5a47b4640 -1 reset not still connected to 0x55c40d3ba750 2024-09-14T05:58:21.001 INFO:tasks.ceph.cluster1.osd.2.smithi143.stderr:2024-09-14T05:58:21.000+0000 7ff5a47b4640 -1 reset not still connected to 0x55c40d3baa90 2024-09-14T05:58:21.002 INFO:tasks.ceph.cluster1.osd.2.smithi143.stderr:2024-09-14T05:58:21.000+0000 7ff5a47b4640 -1 reset not still connected to 0x55c40d3baea0 2024-09-14T05:58:21.002 INFO:tasks.ceph.cluster1.osd.2.smithi143.stderr:2024-09-14T05:58:21.000+0000 7ff5a47b4640 -1 reset not still connected to 0x55c40d69cea0 2024-09-14T05:58:21.002 INFO:tasks.ceph.cluster1.osd.2.smithi143.stderr:2024-09-14T05:58:21.000+0000 7ff5a47b4640 -1 reset not still connected to 0x55c40dd16750 2024-09-14T05:58:21.002 INFO:tasks.ceph.cluster1.osd.2.smithi143.stderr:2024-09-14T05:58:21.000+0000 7ff5a47b4640 -1 reset not still connected to 0x55c40dd16a90 2024-09-14T05:58:21.002 INFO:tasks.ceph.cluster1.osd.2.smithi143.stderr:2024-09-14T05:58:21.000+0000 7ff5a47b4640 -1 reset not still connected to 0x55c40de48680 2024-09-14T05:58:21.002 INFO:tasks.ceph.cluster1.osd.2.smithi143.stderr:2024-09-14T05:58:21.000+0000 7ff5a47b4640 -1 reset not still connected to 0x55c40df94340 2024-09-14T05:58:21.002 INFO:tasks.ceph.cluster1.osd.2.smithi143.stderr:2024-09-14T05:58:21.000+0000 7ff5a47b4640 -1 reset not still connected to 0x55c40df94d00 2024-09-14T05:58:21.002 INFO:tasks.ceph.cluster1.osd.2.smithi143.stderr:2024-09-14T05:58:21.000+0000 7ff5a47b4640 -1 reset not still connected to 0x55c40df95ba0 2024-09-14T05:58:21.002 INFO:tasks.ceph.cluster1.osd.2.smithi143.stderr:2024-09-14T05:58:21.000+0000 7ff5a47b4640 -1 reset not still connected to 0x55c40e5648f0 2024-09-14T05:58:21.002 INFO:tasks.ceph.cluster1.osd.2.smithi143.stderr:2024-09-14T05:58:21.000+0000 7ff5a47b4640 -1 reset not still connected to 0x55c40e564c30 2024-09-14T05:58:21.002 INFO:tasks.ceph.cluster1.osd.2.smithi143.stderr:2024-09-14T05:58:21.000+0000 7ff5a47b4640 -1 reset not still connected to 0x55c412fde0d0 2024-09-14T05:58:21.377 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_40.status 2024-09-14T05:58:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:58:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_40 2024-09-14T05:58:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_40.status 2024-09-14T05:58:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-09-14T05:58:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:58:21.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:58:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=164 2024-09-14T05:58:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 164 = 164 2024-09-14T05:58:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:58:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_40 snap 2024-09-14T05:58:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:58:21.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:21.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_40 2024-09-14T05:58:21.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:58:21.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:58:21.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:58:21.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:58:22.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_40@snap 2024-09-14T05:58:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_40': 2024-09-14T05:58:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:58:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:58:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-14T05:58:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c17f67edff 2024-09-14T05:58:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c17f67edff 2024-09-14T05:58:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:58:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:58:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:58:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:58:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:51:07 2024 2024-09-14T05:58:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:51:07 2024 2024-09-14T05:58:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:51:07 2024 2024-09-14T05:58:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:58:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:58:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:58:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 5049b9ce-c032-4812-91d0-7dbca1b8b176 2024-09-14T05:58:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:58:22.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:58:22.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_40 snap 2024-09-14T05:58:22.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:22.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_40 2024-09-14T05:58:22.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:58:22.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:58:22.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_40.export 2024-09-14T05:58:22.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_40.export 2024-09-14T05:58:22.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_40.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_40.export 2024-09-14T05:58:22.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_40@snap /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_40.export 2024-09-14T05:58:22.658 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:58:22.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_40@snap /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_40.export 2024-09-14T05:58:22.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:58:22.891 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_40.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_40.export 2024-09-14T05:58:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_40.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_40.export 2024-09-14T05:58:22.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:58:22.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:58:22.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_41 2024-09-14T05:58:22.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_41 snap 2024-09-14T05:58:22.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:58:22.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:22.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_41 2024-09-14T05:58:22.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:58:22.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_41@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T05:58:24.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_41 2024-09-14T05:58:24.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:58:24.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:24.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_41 2024-09-14T05:58:24.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_41 started 2024-09-14T05:58:24.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:58:24.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:24.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_41 2024-09-14T05:58:24.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:58:24.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:58:24.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:58:24.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:58:25.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_41 started 2024-09-14T05:58:25.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:58:25.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:25.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_41 2024-09-14T05:58:25.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:58:25.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:58:25.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:58:25.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_41 2024-09-14T05:58:25.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:58:25.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:58:25.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:58:25.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:58:25.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:58:25.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:58:25.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:58:25.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:58:25.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:58:25.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:58:25.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:58:25.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_41 2024-09-14T05:58:25.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:58:25.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:58:25.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:58:25.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:58:25.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:58:25.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:58:25.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:58:25.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:58:25.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:58:25.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:58:25.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:58:25.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:58:25.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:58:25.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:58:25.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:58:25.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:58:25.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:58:25.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:58:25.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:58:25.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:58:25.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:58:25.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:58:25.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:58:25.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:58:25.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:58:25.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:58:25.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:58:25.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:58:25.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:58:25.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:58:25.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:58:25.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:58:25.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:58:25.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_41 2024-09-14T05:58:25.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2024-09-14T05:58:25.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:58:25.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:58:25.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:58:25.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_41 2024-09-14T05:58:25.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-14T05:58:25.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:58:25.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:58:25.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T05:58:25.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-14T05:58:25.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:58:25.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:58:25.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:58:25.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-14T05:58:25.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T05:58:25.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:58:25.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:58:25.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-14T05:58:25.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:58:25.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T05:58:25.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-14T05:58:25.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:58:25.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:58:25.286 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T05:58:25.286 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-14T05:58:25.286 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:58:25.286 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:58:25.286 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:58:25.286 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:58:25.286 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T05:58:25.286 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:58:25.286 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:58:25.286 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-14T05:58:25.286 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-14T05:58:25.286 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T05:58:25.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:58:25.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:58:25.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:58:25.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_41 2024-09-14T05:58:25.375 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2024-09-14T05:58:25.375 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:58:25.375 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:58:25.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:58:25.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_41 2024-09-14T05:58:25.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-14T05:58:25.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:58:25.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:58:25.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T05:58:25.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-14T05:58:25.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:58:25.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:58:25.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:58:25.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-14T05:58:25.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T05:58:25.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:58:25.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:58:25.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-14T05:58:25.380 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:58:25.380 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T05:58:25.380 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-14T05:58:25.380 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:58:25.380 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:58:25.380 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T05:58:25.380 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-14T05:58:25.380 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:58:25.380 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:58:25.380 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:58:25.380 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:58:25.380 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T05:58:25.381 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:58:25.381 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:58:25.381 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-14T05:58:25.381 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-14T05:58:25.381 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T05:58:25.396 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:58:25.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:58:25.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:58:25.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_41 2024-09-14T05:58:25.469 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2024-09-14T05:58:25.469 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:58:25.469 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:58:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:58:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_41 2024-09-14T05:58:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:3 2024-09-14T05:58:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:58:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:58:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-14T05:58:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:3 2024-09-14T05:58:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:58:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:58:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:58:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=3 2024-09-14T05:58:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-14T05:58:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:58:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:58:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=3 2024-09-14T05:58:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-14T05:58:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-14T05:58:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:3 2024-09-14T05:58:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:58:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:58:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-14T05:58:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:3 2024-09-14T05:58:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:58:25.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:58:25.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:58:25.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-14T05:58:25.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-14T05:58:25.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:58:25.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:58:25.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=3 2024-09-14T05:58:25.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=3 2024-09-14T05:58:25.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-14T05:58:25.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T05:58:25.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T05:58:25.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T05:58:25.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_41 2024-09-14T05:58:25.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:58:25.606 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:58:25.606 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:58:25.606 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:58:25.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:58:25.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:58:25.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:58:25.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:58:25.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_41 2024-09-14T05:58:25.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:58:25.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:58:25.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:25.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_41 2024-09-14T05:58:25.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:58:25.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:58:25.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_41 2024-09-14T05:58:25.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:58:25.610 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:58:25.610 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:25.610 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_41 2024-09-14T05:58:25.610 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_41.status 2024-09-14T05:58:25.610 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_41.status 2024-09-14T05:58:25.610 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:58:25.611 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_41.status 2024-09-14T05:58:25.611 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_41.status 2024-09-14T05:58:25.611 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_41.status 2024-09-14T05:58:25.612 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:58:25.612 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_41.status 2024-09-14T05:58:25.612 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_41 2024-09-14T05:58:25.612 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:58:25.612 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:25.612 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_41 2024-09-14T05:58:25.612 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_41 2024-09-14T05:58:26.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 166 2024-09-14T05:58:26.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_41 /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_41.status 2024-09-14T05:58:26.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:58:26.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:26.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_41 2024-09-14T05:58:26.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_41.status 2024-09-14T05:58:26.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_41 --format xml 2024-09-14T05:58:26.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:58:26.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:58:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=166 2024-09-14T05:58:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:58:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:58:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:58:26.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_41.status 2024-09-14T05:58:26.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:58:26.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:26.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_41 2024-09-14T05:58:26.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_41.status 2024-09-14T05:58:26.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2024-09-14T05:58:26.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:58:26.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:58:26.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=75 2024-09-14T05:58:26.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 166 = 75 2024-09-14T05:58:26.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:58:26.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:58:26.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_41.status 2024-09-14T05:58:26.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:58:26.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:26.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_41 2024-09-14T05:58:26.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_41.status 2024-09-14T05:58:26.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2024-09-14T05:58:26.886 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:58:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:58:26.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=75 2024-09-14T05:58:26.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 166 = 75 2024-09-14T05:58:26.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:58:26.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:58:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_41.status 2024-09-14T05:58:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:58:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_41 2024-09-14T05:58:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_41.status 2024-09-14T05:58:27.740 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2024-09-14T05:58:27.740 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:58:27.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:58:27.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:58:27.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:58:27.791 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:58:27.791 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:58:27.791 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:58:27.791 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 166 = '' 2024-09-14T05:58:27.791 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:58:27.791 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-14T05:58:29.393 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_41.status 2024-09-14T05:58:29.393 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:58:29.393 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:29.393 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_41 2024-09-14T05:58:29.393 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_41.status 2024-09-14T05:58:29.393 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2024-09-14T05:58:29.393 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:58:29.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:58:29.450 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=166 2024-09-14T05:58:29.450 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 166 = 166 2024-09-14T05:58:29.450 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:58:29.450 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_41 snap 2024-09-14T05:58:29.450 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:58:29.451 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:29.451 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_41 2024-09-14T05:58:29.451 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:58:29.451 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:58:29.451 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:58:29.451 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:58:30.451 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_41@snap 2024-09-14T05:58:30.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_41': 2024-09-14T05:58:30.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:58:30.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:58:30.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-14T05:58:30.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c119d9944d 2024-09-14T05:58:30.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c119d9944d 2024-09-14T05:58:30.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:58:30.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:58:30.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:58:30.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:58:30.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:51:16 2024 2024-09-14T05:58:30.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:51:16 2024 2024-09-14T05:58:30.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:51:16 2024 2024-09-14T05:58:30.502 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:58:30.502 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:58:30.502 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:58:30.502 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 20523f16-1a82-463b-833f-79e70faaf271 2024-09-14T05:58:30.502 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:58:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:58:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_41 snap 2024-09-14T05:58:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_41 2024-09-14T05:58:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:58:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:58:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_41.export 2024-09-14T05:58:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_41.export 2024-09-14T05:58:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_41.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_41.export 2024-09-14T05:58:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_41@snap /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_41.export 2024-09-14T05:58:30.677 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:58:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_41@snap /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_41.export 2024-09-14T05:58:30.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:58:30.890 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_41.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_41.export 2024-09-14T05:58:30.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_41.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_41.export 2024-09-14T05:58:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:58:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:58:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_42 2024-09-14T05:58:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_42 snap 2024-09-14T05:58:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:58:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_42 2024-09-14T05:58:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:58:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_42@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T05:58:32.193 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_42 2024-09-14T05:58:32.193 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:58:32.193 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:32.193 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_42 2024-09-14T05:58:32.193 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_42 started 2024-09-14T05:58:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:58:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_42 2024-09-14T05:58:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:58:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:58:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:58:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:58:33.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_42 started 2024-09-14T05:58:33.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:58:33.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:33.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_42 2024-09-14T05:58:33.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:58:33.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:58:33.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:58:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_42 2024-09-14T05:58:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:58:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:58:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:58:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:58:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:58:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:58:33.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:58:33.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:58:33.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:58:33.198 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:58:33.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:58:33.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_42 2024-09-14T05:58:33.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:58:33.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:58:33.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:58:33.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:58:33.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:58:33.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:58:33.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:58:33.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:58:33.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:58:33.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:58:33.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:58:33.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:58:33.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:58:33.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:58:33.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:58:33.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:58:33.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:58:33.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:58:33.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:58:33.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:58:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:58:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:58:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:58:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:58:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:58:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:58:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:58:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:58:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:58:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:58:33.221 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:58:33.221 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:58:33.221 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:58:33.221 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_42 2024-09-14T05:58:33.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2024-09-14T05:58:33.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:58:33.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:58:33.301 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:58:33.301 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_42 2024-09-14T05:58:33.302 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-14T05:58:33.302 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:58:33.302 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:58:33.302 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T05:58:33.302 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-14T05:58:33.302 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:58:33.302 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:58:33.302 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:58:33.302 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-14T05:58:33.302 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T05:58:33.302 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:58:33.302 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:58:33.302 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-14T05:58:33.302 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:58:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T05:58:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-14T05:58:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:58:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:58:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T05:58:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-14T05:58:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:58:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:58:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:58:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:58:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T05:58:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:58:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:58:33.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-14T05:58:33.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-14T05:58:33.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T05:58:33.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:58:33.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:58:33.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:58:33.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_42 2024-09-14T05:58:33.393 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2024-09-14T05:58:33.393 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:58:33.393 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:58:33.396 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:58:33.396 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_42 2024-09-14T05:58:33.396 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-14T05:58:33.396 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:58:33.396 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:58:33.396 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T05:58:33.396 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-14T05:58:33.396 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:58:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:58:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:58:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-14T05:58:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T05:58:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:58:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:58:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-14T05:58:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:58:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T05:58:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-14T05:58:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:58:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:58:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T05:58:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-14T05:58:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:58:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:58:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:58:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:58:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T05:58:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:58:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:58:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-14T05:58:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-14T05:58:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T05:58:33.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:58:33.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:58:33.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:58:33.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_42 2024-09-14T05:58:33.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2024-09-14T05:58:33.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:58:33.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:58:33.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:58:33.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_42 2024-09-14T05:58:33.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:3 2024-09-14T05:58:33.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:58:33.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:58:33.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-14T05:58:33.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:3 2024-09-14T05:58:33.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:58:33.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:58:33.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:58:33.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=3 2024-09-14T05:58:33.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-14T05:58:33.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:58:33.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:58:33.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=3 2024-09-14T05:58:33.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-14T05:58:33.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-14T05:58:33.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:3 2024-09-14T05:58:33.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:58:33.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:58:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-14T05:58:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:3 2024-09-14T05:58:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:58:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:58:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:58:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-14T05:58:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-14T05:58:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:58:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:58:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=3 2024-09-14T05:58:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=3 2024-09-14T05:58:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-14T05:58:33.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T05:58:33.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T05:58:33.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T05:58:33.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_42 2024-09-14T05:58:33.624 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:58:33.624 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:58:33.624 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:58:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:58:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:58:33.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:58:33.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:58:33.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:58:33.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_42 2024-09-14T05:58:33.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:58:33.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:58:33.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:33.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_42 2024-09-14T05:58:33.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:58:33.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:58:33.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_42 2024-09-14T05:58:33.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:58:33.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:58:33.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:33.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_42 2024-09-14T05:58:33.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_42.status 2024-09-14T05:58:33.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_42.status 2024-09-14T05:58:33.628 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:58:33.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_42.status 2024-09-14T05:58:33.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_42.status 2024-09-14T05:58:33.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_42.status 2024-09-14T05:58:33.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:58:33.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_42.status 2024-09-14T05:58:33.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_42 2024-09-14T05:58:33.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:58:33.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:33.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_42 2024-09-14T05:58:33.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_42 2024-09-14T05:58:34.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 168 2024-09-14T05:58:34.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_42 /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_42.status 2024-09-14T05:58:34.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:58:34.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:34.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_42 2024-09-14T05:58:34.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_42.status 2024-09-14T05:58:34.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_42 --format xml 2024-09-14T05:58:34.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:58:34.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:58:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=168 2024-09-14T05:58:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:58:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:58:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:58:34.452 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_42.status 2024-09-14T05:58:34.452 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:58:34.452 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:34.452 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_42 2024-09-14T05:58:34.452 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_42.status 2024-09-14T05:58:34.452 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2024-09-14T05:58:34.452 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:58:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:58:34.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=76 2024-09-14T05:58:34.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 168 = 76 2024-09-14T05:58:34.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:58:34.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:58:34.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_42.status 2024-09-14T05:58:34.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:58:34.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:34.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_42 2024-09-14T05:58:34.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_42.status 2024-09-14T05:58:34.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2024-09-14T05:58:34.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:58:34.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:58:34.956 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=76 2024-09-14T05:58:34.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 168 = 76 2024-09-14T05:58:34.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:58:34.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:58:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_42.status 2024-09-14T05:58:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:58:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_42 2024-09-14T05:58:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_42.status 2024-09-14T05:58:35.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2024-09-14T05:58:35.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:58:35.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:58:35.811 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:58:35.812 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:58:35.812 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:58:35.812 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:58:35.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:58:35.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 168 = '' 2024-09-14T05:58:35.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:58:35.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-14T05:58:37.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_42.status 2024-09-14T05:58:37.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:58:37.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:37.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_42 2024-09-14T05:58:37.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_42.status 2024-09-14T05:58:37.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2024-09-14T05:58:37.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:58:37.469 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:58:37.472 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=168 2024-09-14T05:58:37.472 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 168 = 168 2024-09-14T05:58:37.472 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:58:37.472 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_42 snap 2024-09-14T05:58:37.472 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:58:37.472 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:37.472 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_42 2024-09-14T05:58:37.472 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:58:37.472 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:58:37.472 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:58:37.472 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:58:38.472 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_42@snap 2024-09-14T05:58:38.524 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_42': 2024-09-14T05:58:38.524 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:58:38.524 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:58:38.524 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-14T05:58:38.524 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c16b56ff13 2024-09-14T05:58:38.524 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c16b56ff13 2024-09-14T05:58:38.524 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:58:38.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:58:38.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:58:38.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:58:38.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:51:26 2024 2024-09-14T05:58:38.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:51:26 2024 2024-09-14T05:58:38.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:51:26 2024 2024-09-14T05:58:38.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:58:38.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:58:38.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:58:38.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 662fa768-1dea-4f15-91bf-edf297ebd84a 2024-09-14T05:58:38.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:58:38.528 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:58:38.528 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_42 snap 2024-09-14T05:58:38.528 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:38.529 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_42 2024-09-14T05:58:38.529 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:58:38.529 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:58:38.529 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_42.export 2024-09-14T05:58:38.529 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_42.export 2024-09-14T05:58:38.529 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_42.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_42.export 2024-09-14T05:58:38.529 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_42@snap /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_42.export 2024-09-14T05:58:38.699 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:58:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_42@snap /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_42.export 2024-09-14T05:58:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:58:38.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_42.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_42.export 2024-09-14T05:58:39.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_42.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_42.export 2024-09-14T05:58:39.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:58:39.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:58:39.035 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_43 2024-09-14T05:58:39.035 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_43 snap 2024-09-14T05:58:39.035 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:58:39.035 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:39.035 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_43 2024-09-14T05:58:39.035 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:58:39.035 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_43@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T05:58:41.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_43 2024-09-14T05:58:41.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:58:41.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:41.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_43 2024-09-14T05:58:41.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_43 started 2024-09-14T05:58:41.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:58:41.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:41.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_43 2024-09-14T05:58:41.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:58:41.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:58:41.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:58:41.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:58:42.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_43 started 2024-09-14T05:58:42.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:58:42.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:42.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_43 2024-09-14T05:58:42.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:58:42.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:58:42.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:58:42.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_43 2024-09-14T05:58:42.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:58:42.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:58:42.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:58:42.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:58:42.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:58:42.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:58:42.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:58:42.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:58:42.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:58:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:58:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:58:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_43 2024-09-14T05:58:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:58:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:58:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:58:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:58:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:58:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:58:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:58:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:58:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:58:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:58:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:58:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:58:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:58:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:58:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:58:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:58:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:58:42.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:58:42.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:58:42.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:58:42.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:58:42.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:58:42.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:58:42.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:58:42.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:58:42.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:58:42.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:58:42.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:58:42.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:58:42.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:58:42.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:58:42.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:58:42.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:58:42.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_43 2024-09-14T05:58:42.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2024-09-14T05:58:42.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:58:42.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:58:42.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:58:42.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_43 2024-09-14T05:58:42.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-14T05:58:42.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:58:42.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:58:42.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T05:58:42.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-14T05:58:42.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:58:42.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:58:42.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:58:42.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-14T05:58:42.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T05:58:42.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:58:42.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:58:42.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-14T05:58:42.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:58:42.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T05:58:42.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-14T05:58:42.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:58:42.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:58:42.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T05:58:42.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-14T05:58:42.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:58:42.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:58:42.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:58:42.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:58:42.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T05:58:42.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:58:42.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:58:42.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-14T05:58:42.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-14T05:58:42.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T05:58:42.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:58:42.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:58:42.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:58:42.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_43 2024-09-14T05:58:42.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2024-09-14T05:58:42.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:58:42.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:58:42.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:58:42.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_43 2024-09-14T05:58:42.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-14T05:58:42.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:58:42.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:58:42.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T05:58:42.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-14T05:58:42.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:58:42.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:58:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:58:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-14T05:58:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T05:58:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:58:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:58:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-14T05:58:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:58:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T05:58:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-14T05:58:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:58:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:58:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T05:58:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-14T05:58:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:58:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:58:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:58:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:58:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T05:58:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:58:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:58:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-14T05:58:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-14T05:58:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T05:58:42.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:58:42.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:58:42.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:58:42.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_43 2024-09-14T05:58:42.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2024-09-14T05:58:42.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:58:42.506 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:58:42.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:58:42.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_43 2024-09-14T05:58:42.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:3 2024-09-14T05:58:42.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:58:42.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:58:42.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-14T05:58:42.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:3 2024-09-14T05:58:42.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:58:42.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:58:42.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:58:42.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=3 2024-09-14T05:58:42.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-14T05:58:42.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:58:42.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:58:42.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=3 2024-09-14T05:58:42.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-14T05:58:42.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-14T05:58:42.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:3 2024-09-14T05:58:42.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:58:42.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:58:42.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-14T05:58:42.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:3 2024-09-14T05:58:42.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:58:42.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:58:42.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:58:42.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-14T05:58:42.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-14T05:58:42.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:58:42.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:58:42.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=3 2024-09-14T05:58:42.510 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=3 2024-09-14T05:58:42.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-14T05:58:42.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T05:58:42.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T05:58:42.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T05:58:42.528 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_43 2024-09-14T05:58:42.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:58:42.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:58:42.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:58:42.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:58:42.645 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:58:42.646 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:58:42.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:58:42.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:58:42.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_43 2024-09-14T05:58:42.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:58:42.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:58:42.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:42.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_43 2024-09-14T05:58:42.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:58:42.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:58:42.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_43 2024-09-14T05:58:42.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:58:42.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:58:42.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:42.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_43 2024-09-14T05:58:42.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_43.status 2024-09-14T05:58:42.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_43.status 2024-09-14T05:58:42.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:58:42.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_43.status 2024-09-14T05:58:42.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_43.status 2024-09-14T05:58:42.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_43.status 2024-09-14T05:58:42.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:58:42.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_43.status 2024-09-14T05:58:42.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_43 2024-09-14T05:58:42.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:58:42.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:42.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_43 2024-09-14T05:58:42.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_43 2024-09-14T05:58:43.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 170 2024-09-14T05:58:43.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_43 /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_43.status 2024-09-14T05:58:43.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:58:43.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:43.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_43 2024-09-14T05:58:43.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_43.status 2024-09-14T05:58:43.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_43 --format xml 2024-09-14T05:58:43.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:58:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:58:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=170 2024-09-14T05:58:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:58:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:58:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:58:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_43.status 2024-09-14T05:58:43.468 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:58:43.468 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:43.468 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_43 2024-09-14T05:58:43.468 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_43.status 2024-09-14T05:58:43.468 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2024-09-14T05:58:43.468 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:58:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:58:43.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=77 2024-09-14T05:58:43.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 170 = 77 2024-09-14T05:58:43.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:58:43.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:58:43.920 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_43.status 2024-09-14T05:58:43.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:58:43.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:43.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_43 2024-09-14T05:58:43.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_43.status 2024-09-14T05:58:43.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2024-09-14T05:58:43.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:58:43.969 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:58:43.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=77 2024-09-14T05:58:43.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 170 = 77 2024-09-14T05:58:43.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:58:43.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:58:44.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_43.status 2024-09-14T05:58:44.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:58:44.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:44.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_43 2024-09-14T05:58:44.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_43.status 2024-09-14T05:58:44.773 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2024-09-14T05:58:44.773 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:58:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:58:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:58:44.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:58:44.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:58:44.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:58:44.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:58:44.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 170 = '' 2024-09-14T05:58:44.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:58:44.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-14T05:58:46.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_43.status 2024-09-14T05:58:46.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:58:46.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:46.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_43 2024-09-14T05:58:46.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_43.status 2024-09-14T05:58:46.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2024-09-14T05:58:46.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:58:46.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:58:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=170 2024-09-14T05:58:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 170 = 170 2024-09-14T05:58:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:58:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_43 snap 2024-09-14T05:58:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:58:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_43 2024-09-14T05:58:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:58:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:58:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:58:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:58:47.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_43@snap 2024-09-14T05:58:47.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_43': 2024-09-14T05:58:47.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:58:47.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:58:47.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-14T05:58:47.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c18724011d 2024-09-14T05:58:47.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c18724011d 2024-09-14T05:58:47.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:58:47.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:58:47.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:58:47.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:58:47.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:51:36 2024 2024-09-14T05:58:47.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:51:36 2024 2024-09-14T05:58:47.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:51:36 2024 2024-09-14T05:58:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:58:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:58:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:58:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: f71c9e7c-cbd0-4274-bae7-2c98cc5052b1 2024-09-14T05:58:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:58:47.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:58:47.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_43 snap 2024-09-14T05:58:47.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:47.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_43 2024-09-14T05:58:47.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:58:47.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:58:47.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_43.export 2024-09-14T05:58:47.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_43.export 2024-09-14T05:58:47.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_43.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_43.export 2024-09-14T05:58:47.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_43@snap /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_43.export 2024-09-14T05:58:47.709 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:58:47.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_43@snap /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_43.export 2024-09-14T05:58:47.948 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:58:47.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_43.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_43.export 2024-09-14T05:58:48.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_43.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_43.export 2024-09-14T05:58:48.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:58:48.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:58:48.040 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_44 2024-09-14T05:58:48.040 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_44 snap 2024-09-14T05:58:48.040 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:58:48.040 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:48.040 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_44 2024-09-14T05:58:48.041 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:58:48.041 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_44@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T05:58:49.229 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_44 2024-09-14T05:58:49.229 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:58:49.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:49.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_44 2024-09-14T05:58:49.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_44 started 2024-09-14T05:58:49.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:58:49.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:49.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_44 2024-09-14T05:58:49.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:58:49.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:58:49.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:58:49.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:58:50.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_44 started 2024-09-14T05:58:50.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:58:50.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:50.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_44 2024-09-14T05:58:50.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:58:50.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:58:50.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:58:50.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_44 2024-09-14T05:58:50.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:58:50.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:58:50.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:58:50.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:58:50.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:58:50.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:58:50.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:58:50.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:58:50.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:58:50.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:58:50.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:58:50.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_44 2024-09-14T05:58:50.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:58:50.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:58:50.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:58:50.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:58:50.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:58:50.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:58:50.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:58:50.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:58:50.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:58:50.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:58:50.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:58:50.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:58:50.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:58:50.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:58:50.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:58:50.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:58:50.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:58:50.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:58:50.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:58:50.237 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:58:50.237 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:58:50.237 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:58:50.237 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:58:50.237 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:58:50.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:58:50.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:58:50.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:58:50.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:58:50.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:58:50.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:58:50.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:58:50.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:58:50.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:58:50.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_44 2024-09-14T05:58:50.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2024-09-14T05:58:50.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:58:50.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:58:50.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:58:50.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_44 2024-09-14T05:58:50.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-14T05:58:50.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:58:50.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:58:50.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T05:58:50.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-14T05:58:50.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:58:50.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:58:50.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:58:50.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-14T05:58:50.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T05:58:50.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:58:50.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:58:50.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-14T05:58:50.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:58:50.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T05:58:50.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-14T05:58:50.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:58:50.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:58:50.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T05:58:50.340 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-14T05:58:50.340 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:58:50.340 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:58:50.340 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:58:50.340 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:58:50.340 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T05:58:50.340 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:58:50.340 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:58:50.340 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-14T05:58:50.340 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-14T05:58:50.340 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T05:58:50.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:58:50.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:58:50.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:58:50.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_44 2024-09-14T05:58:50.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2024-09-14T05:58:50.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:58:50.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:58:50.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:58:50.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_44 2024-09-14T05:58:50.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-14T05:58:50.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:58:50.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:58:50.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T05:58:50.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-14T05:58:50.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:58:50.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:58:50.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:58:50.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-14T05:58:50.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T05:58:50.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:58:50.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:58:50.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-14T05:58:50.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:58:50.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T05:58:50.433 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-14T05:58:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:58:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:58:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T05:58:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-14T05:58:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:58:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:58:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:58:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:58:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T05:58:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:58:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:58:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-14T05:58:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-14T05:58:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T05:58:50.450 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:58:50.450 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:58:50.450 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:58:50.450 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_44 2024-09-14T05:58:50.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2024-09-14T05:58:50.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:58:50.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:58:50.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:58:50.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_44 2024-09-14T05:58:50.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:3 2024-09-14T05:58:50.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:58:50.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:58:50.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-14T05:58:50.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:3 2024-09-14T05:58:50.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:58:50.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:58:50.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:58:50.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=3 2024-09-14T05:58:50.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-14T05:58:50.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:58:50.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:58:50.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=3 2024-09-14T05:58:50.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-14T05:58:50.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-14T05:58:50.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:3 2024-09-14T05:58:50.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:58:50.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:58:50.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-14T05:58:50.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:3 2024-09-14T05:58:50.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:58:50.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:58:50.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:58:50.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-14T05:58:50.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-14T05:58:50.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:58:50.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:58:50.528 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=3 2024-09-14T05:58:50.528 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=3 2024-09-14T05:58:50.528 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-14T05:58:50.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T05:58:50.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T05:58:50.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T05:58:50.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_44 2024-09-14T05:58:50.659 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:58:50.659 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:58:50.659 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:58:50.659 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:58:50.660 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:58:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:58:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:58:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:58:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_44 2024-09-14T05:58:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:58:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:58:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_44 2024-09-14T05:58:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:58:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:58:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_44 2024-09-14T05:58:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:58:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:58:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_44 2024-09-14T05:58:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_44.status 2024-09-14T05:58:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_44.status 2024-09-14T05:58:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:58:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_44.status 2024-09-14T05:58:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_44.status 2024-09-14T05:58:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_44.status 2024-09-14T05:58:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:58:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_44.status 2024-09-14T05:58:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_44 2024-09-14T05:58:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:58:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_44 2024-09-14T05:58:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_44 2024-09-14T05:58:51.229 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 172 2024-09-14T05:58:51.250 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_44 /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_44.status 2024-09-14T05:58:51.250 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:58:51.250 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:51.250 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_44 2024-09-14T05:58:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_44.status 2024-09-14T05:58:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_44 --format xml 2024-09-14T05:58:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:58:51.286 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:58:51.288 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=172 2024-09-14T05:58:51.288 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:58:51.288 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:58:51.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:58:51.489 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_44.status 2024-09-14T05:58:51.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:58:51.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:51.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_44 2024-09-14T05:58:51.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_44.status 2024-09-14T05:58:51.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2024-09-14T05:58:51.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:58:51.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:58:51.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=78 2024-09-14T05:58:51.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 172 = 78 2024-09-14T05:58:51.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:58:51.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:58:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_44.status 2024-09-14T05:58:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:58:51.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:51.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_44 2024-09-14T05:58:51.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_44.status 2024-09-14T05:58:51.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2024-09-14T05:58:51.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:58:51.991 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:58:51.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=78 2024-09-14T05:58:51.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 172 = 78 2024-09-14T05:58:51.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:58:51.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:58:52.795 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_44.status 2024-09-14T05:58:52.795 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:58:52.795 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:52.795 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_44 2024-09-14T05:58:52.795 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_44.status 2024-09-14T05:58:52.795 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2024-09-14T05:58:52.795 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:58:52.847 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:58:52.848 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:58:52.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:58:52.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:58:52.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:58:52.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:58:52.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 172 = '' 2024-09-14T05:58:52.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:58:52.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-14T05:58:54.451 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_44.status 2024-09-14T05:58:54.451 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:58:54.451 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:54.451 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_44 2024-09-14T05:58:54.451 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_44.status 2024-09-14T05:58:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2024-09-14T05:58:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:58:54.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:58:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=172 2024-09-14T05:58:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 172 = 172 2024-09-14T05:58:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:58:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_44 snap 2024-09-14T05:58:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:58:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_44 2024-09-14T05:58:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:58:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:58:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:58:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:58:55.506 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_44@snap 2024-09-14T05:58:55.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_44': 2024-09-14T05:58:55.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:58:55.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:58:55.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-14T05:58:55.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c17bdb5b82 2024-09-14T05:58:55.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c17bdb5b82 2024-09-14T05:58:55.556 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:58:55.556 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:58:55.556 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:58:55.556 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:58:55.556 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:51:46 2024 2024-09-14T05:58:55.556 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:51:46 2024 2024-09-14T05:58:55.556 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:51:46 2024 2024-09-14T05:58:55.556 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:58:55.556 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:58:55.556 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:58:55.556 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 40153520-d46c-4ade-9ebf-442136d19014 2024-09-14T05:58:55.556 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:58:55.559 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:58:55.559 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_44 snap 2024-09-14T05:58:55.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:55.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_44 2024-09-14T05:58:55.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:58:55.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:58:55.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_44.export 2024-09-14T05:58:55.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_44.export 2024-09-14T05:58:55.560 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_44.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_44.export 2024-09-14T05:58:55.561 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_44@snap /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_44.export 2024-09-14T05:58:55.734 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:58:55.741 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_44@snap /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_44.export 2024-09-14T05:58:55.967 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:58:55.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_44.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_44.export 2024-09-14T05:58:56.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_44.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_44.export 2024-09-14T05:58:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:58:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:58:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_45 2024-09-14T05:58:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_45 snap 2024-09-14T05:58:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:58:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_45 2024-09-14T05:58:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:58:56.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_45@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T05:58:58.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_45 2024-09-14T05:58:58.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:58:58.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:58.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_45 2024-09-14T05:58:58.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_45 started 2024-09-14T05:58:58.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:58:58.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:58.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_45 2024-09-14T05:58:58.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:58:58.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:58:58.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:58:58.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:58:59.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_45 started 2024-09-14T05:58:59.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:58:59.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:59.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_45 2024-09-14T05:58:59.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:58:59.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:58:59.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:58:59.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_45 2024-09-14T05:58:59.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:58:59.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:58:59.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:58:59.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:58:59.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:58:59.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:58:59.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:58:59.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:58:59.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:58:59.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:58:59.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:58:59.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_45 2024-09-14T05:58:59.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:58:59.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:58:59.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:58:59.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:58:59.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:58:59.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:58:59.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:58:59.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:58:59.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:58:59.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:58:59.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:58:59.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:58:59.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:58:59.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:58:59.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:58:59.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:58:59.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:58:59.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:58:59.151 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:58:59.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:58:59.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:58:59.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:58:59.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:58:59.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:58:59.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:58:59.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:58:59.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:58:59.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:58:59.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:58:59.152 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:58:59.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:58:59.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:58:59.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:58:59.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_45 2024-09-14T05:58:59.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-09-14T05:58:59.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:58:59.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:58:59.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:58:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_45 2024-09-14T05:58:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-14T05:58:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:58:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:58:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T05:58:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-14T05:58:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:58:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:58:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:58:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-14T05:58:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T05:58:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:58:59.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:58:59.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-14T05:58:59.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:58:59.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T05:58:59.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-14T05:58:59.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:58:59.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:58:59.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T05:58:59.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-14T05:58:59.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:58:59.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:58:59.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:58:59.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:58:59.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T05:58:59.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:58:59.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:58:59.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-14T05:58:59.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-14T05:58:59.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T05:58:59.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:58:59.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:58:59.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:58:59.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_45 2024-09-14T05:58:59.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-09-14T05:58:59.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:58:59.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:58:59.352 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:58:59.352 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_45 2024-09-14T05:58:59.352 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-14T05:58:59.352 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:58:59.352 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:58:59.352 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T05:58:59.352 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-14T05:58:59.352 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:58:59.352 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:58:59.352 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:58:59.352 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-14T05:58:59.352 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T05:58:59.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:58:59.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:58:59.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-14T05:58:59.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:58:59.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T05:58:59.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-14T05:58:59.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:58:59.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:58:59.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T05:58:59.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-14T05:58:59.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:58:59.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:58:59.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:58:59.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:58:59.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T05:58:59.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:58:59.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:58:59.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-14T05:58:59.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-14T05:58:59.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T05:58:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:58:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:58:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:58:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_45 2024-09-14T05:58:59.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-09-14T05:58:59.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:58:59.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:58:59.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:58:59.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_45 2024-09-14T05:58:59.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:3 2024-09-14T05:58:59.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:58:59.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:58:59.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-14T05:58:59.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:3 2024-09-14T05:58:59.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:58:59.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:58:59.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:58:59.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=3 2024-09-14T05:58:59.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-14T05:58:59.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:58:59.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:58:59.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=3 2024-09-14T05:58:59.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-14T05:58:59.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-14T05:58:59.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:3 2024-09-14T05:58:59.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:58:59.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:58:59.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-14T05:58:59.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:3 2024-09-14T05:58:59.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:58:59.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:58:59.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:58:59.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-14T05:58:59.450 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-14T05:58:59.450 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:58:59.450 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:58:59.450 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=3 2024-09-14T05:58:59.450 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=3 2024-09-14T05:58:59.450 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-14T05:58:59.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T05:58:59.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T05:58:59.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T05:58:59.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_45 2024-09-14T05:58:59.584 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:58:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:58:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:58:59.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:58:59.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:58:59.588 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:58:59.588 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:58:59.588 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:58:59.588 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_45 2024-09-14T05:58:59.588 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:58:59.588 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:58:59.588 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:59.588 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_45 2024-09-14T05:58:59.588 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:58:59.589 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:58:59.589 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_45 2024-09-14T05:58:59.589 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:58:59.589 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:58:59.589 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:59.589 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_45 2024-09-14T05:58:59.589 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_45.status 2024-09-14T05:58:59.589 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_45.status 2024-09-14T05:58:59.589 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:58:59.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_45.status 2024-09-14T05:58:59.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_45.status 2024-09-14T05:58:59.590 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_45.status 2024-09-14T05:58:59.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:58:59.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_45.status 2024-09-14T05:58:59.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_45 2024-09-14T05:58:59.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:58:59.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:58:59.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_45 2024-09-14T05:58:59.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_45 2024-09-14T05:59:00.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 174 2024-09-14T05:59:00.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_45 /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_45.status 2024-09-14T05:59:00.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:59:00.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:00.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_45 2024-09-14T05:59:00.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_45.status 2024-09-14T05:59:00.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_45 --format xml 2024-09-14T05:59:00.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:59:00.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:59:00.198 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=174 2024-09-14T05:59:00.198 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:59:00.198 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:59:00.198 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:59:00.398 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_45.status 2024-09-14T05:59:00.398 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:59:00.399 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:00.399 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_45 2024-09-14T05:59:00.399 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_45.status 2024-09-14T05:59:00.399 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2024-09-14T05:59:00.399 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:59:00.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:59:00.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=79 2024-09-14T05:59:00.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 174 = 79 2024-09-14T05:59:00.450 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:59:00.450 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:59:00.851 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_45.status 2024-09-14T05:59:00.851 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:59:00.851 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:00.851 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_45 2024-09-14T05:59:00.851 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_45.status 2024-09-14T05:59:00.851 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2024-09-14T05:59:00.851 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:59:00.900 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:59:00.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=79 2024-09-14T05:59:00.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 174 = 79 2024-09-14T05:59:00.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:59:00.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:59:01.704 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_45.status 2024-09-14T05:59:01.704 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:59:01.704 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:01.704 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_45 2024-09-14T05:59:01.704 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_45.status 2024-09-14T05:59:01.704 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2024-09-14T05:59:01.705 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:59:01.756 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:59:01.756 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:59:01.758 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:59:01.758 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:59:01.758 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:59:01.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:59:01.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 174 = '' 2024-09-14T05:59:01.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:59:01.759 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-14T05:59:03.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_45.status 2024-09-14T05:59:03.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:59:03.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:03.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_45 2024-09-14T05:59:03.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_45.status 2024-09-14T05:59:03.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2024-09-14T05:59:03.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:59:03.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:59:03.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=174 2024-09-14T05:59:03.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 174 = 174 2024-09-14T05:59:03.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:59:03.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_45 snap 2024-09-14T05:59:03.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:59:03.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:03.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_45 2024-09-14T05:59:03.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:59:03.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:59:03.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:59:03.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:59:04.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_45@snap 2024-09-14T05:59:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_45': 2024-09-14T05:59:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:59:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:59:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-14T05:59:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c1c606459d 2024-09-14T05:59:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c1c606459d 2024-09-14T05:59:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:59:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:59:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:59:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:59:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:51:57 2024 2024-09-14T05:59:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:51:57 2024 2024-09-14T05:59:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:51:57 2024 2024-09-14T05:59:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:59:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:59:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:59:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 8ed3596b-4b0c-421c-ae91-4a8f4c37bca1 2024-09-14T05:59:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:59:04.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:59:04.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_45 snap 2024-09-14T05:59:04.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:04.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_45 2024-09-14T05:59:04.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:59:04.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:59:04.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_45.export 2024-09-14T05:59:04.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_45.export 2024-09-14T05:59:04.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_45.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_45.export 2024-09-14T05:59:04.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_45@snap /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_45.export 2024-09-14T05:59:04.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:59:04.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_45@snap /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_45.export 2024-09-14T05:59:04.899 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:59:04.905 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_45.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_45.export 2024-09-14T05:59:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_45.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_45.export 2024-09-14T05:59:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:59:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:59:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_46 2024-09-14T05:59:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_46 snap 2024-09-14T05:59:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:59:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_46 2024-09-14T05:59:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:59:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_46@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T05:59:06.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_46 2024-09-14T05:59:06.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:59:06.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:06.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_46 2024-09-14T05:59:06.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_46 started 2024-09-14T05:59:06.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:59:06.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:06.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_46 2024-09-14T05:59:06.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:59:06.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:59:06.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:59:06.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:59:07.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_46 started 2024-09-14T05:59:07.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:59:07.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:07.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_46 2024-09-14T05:59:07.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:59:07.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:59:07.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:59:07.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_46 2024-09-14T05:59:07.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:59:07.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:59:07.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:59:07.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:59:07.171 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:59:07.171 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:59:07.171 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:59:07.171 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:59:07.171 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:59:07.171 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:59:07.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:59:07.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_46 2024-09-14T05:59:07.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:59:07.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:59:07.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:59:07.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:59:07.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:59:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:59:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:59:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:59:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:59:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:59:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:59:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:59:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:59:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:59:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:59:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:59:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:59:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:59:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:59:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:59:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:59:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:59:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:59:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:59:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:59:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:59:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:59:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:59:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:59:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:59:07.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:59:07.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:59:07.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:59:07.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_46 2024-09-14T05:59:07.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-09-14T05:59:07.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:59:07.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:59:07.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:59:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_46 2024-09-14T05:59:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-14T05:59:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:59:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:59:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T05:59:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-14T05:59:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:59:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:59:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:59:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-14T05:59:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T05:59:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:59:07.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:59:07.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-14T05:59:07.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:59:07.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T05:59:07.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-14T05:59:07.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:59:07.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:59:07.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T05:59:07.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-14T05:59:07.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:59:07.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:59:07.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:59:07.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:59:07.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T05:59:07.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:59:07.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:59:07.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-14T05:59:07.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-14T05:59:07.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T05:59:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:59:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:59:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:59:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_46 2024-09-14T05:59:07.373 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-09-14T05:59:07.373 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:59:07.373 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:59:07.377 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:59:07.377 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_46 2024-09-14T05:59:07.377 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-14T05:59:07.377 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:59:07.377 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:59:07.377 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T05:59:07.377 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-14T05:59:07.377 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:59:07.377 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:59:07.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:59:07.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-14T05:59:07.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T05:59:07.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:59:07.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:59:07.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-14T05:59:07.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:59:07.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T05:59:07.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-14T05:59:07.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:59:07.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:59:07.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T05:59:07.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-14T05:59:07.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:59:07.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:59:07.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:59:07.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:59:07.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T05:59:07.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:59:07.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:59:07.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-14T05:59:07.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-14T05:59:07.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T05:59:07.396 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:59:07.396 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:59:07.396 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:59:07.396 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_46 2024-09-14T05:59:07.469 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-09-14T05:59:07.469 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:59:07.469 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:59:07.472 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:59:07.472 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_46 2024-09-14T05:59:07.472 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:3 2024-09-14T05:59:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:59:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:59:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-14T05:59:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:3 2024-09-14T05:59:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:59:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:59:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:59:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=3 2024-09-14T05:59:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-14T05:59:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:59:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:59:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=3 2024-09-14T05:59:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-14T05:59:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-14T05:59:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:3 2024-09-14T05:59:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:59:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:59:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-14T05:59:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:3 2024-09-14T05:59:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:59:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:59:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:59:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-14T05:59:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-14T05:59:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:59:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:59:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=3 2024-09-14T05:59:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=3 2024-09-14T05:59:07.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-14T05:59:07.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T05:59:07.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T05:59:07.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T05:59:07.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_46 2024-09-14T05:59:07.611 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:59:07.612 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:59:07.612 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:59:07.612 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:59:07.613 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:59:07.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:59:07.614 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:59:07.615 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:59:07.615 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_46 2024-09-14T05:59:07.615 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:59:07.615 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:59:07.615 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:07.615 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_46 2024-09-14T05:59:07.615 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:59:07.615 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:59:07.615 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_46 2024-09-14T05:59:07.615 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:59:07.615 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:59:07.615 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:07.615 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_46 2024-09-14T05:59:07.615 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_46.status 2024-09-14T05:59:07.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_46.status 2024-09-14T05:59:07.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:59:07.616 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_46.status 2024-09-14T05:59:07.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_46.status 2024-09-14T05:59:07.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_46.status 2024-09-14T05:59:07.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:59:07.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_46.status 2024-09-14T05:59:07.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_46 2024-09-14T05:59:07.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:59:07.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:07.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_46 2024-09-14T05:59:07.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_46 2024-09-14T05:59:08.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 176 2024-09-14T05:59:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_46 /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_46.status 2024-09-14T05:59:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:59:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_46 2024-09-14T05:59:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_46.status 2024-09-14T05:59:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_46 --format xml 2024-09-14T05:59:08.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:59:08.225 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:59:08.227 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=176 2024-09-14T05:59:08.227 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:59:08.227 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:59:08.227 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:59:08.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_46.status 2024-09-14T05:59:08.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:59:08.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:08.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_46 2024-09-14T05:59:08.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_46.status 2024-09-14T05:59:08.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2024-09-14T05:59:08.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:59:08.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:59:08.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=80 2024-09-14T05:59:08.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 176 = 80 2024-09-14T05:59:08.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:59:08.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:59:08.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_46.status 2024-09-14T05:59:08.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:59:08.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:08.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_46 2024-09-14T05:59:08.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_46.status 2024-09-14T05:59:08.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2024-09-14T05:59:08.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:59:08.931 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:59:08.932 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=80 2024-09-14T05:59:08.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 176 = 80 2024-09-14T05:59:08.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:59:08.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:59:09.734 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_46.status 2024-09-14T05:59:09.734 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:59:09.734 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:09.734 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_46 2024-09-14T05:59:09.734 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_46.status 2024-09-14T05:59:09.734 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2024-09-14T05:59:09.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:59:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:59:09.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:59:09.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:59:09.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:59:09.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:59:09.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:59:09.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 176 = '' 2024-09-14T05:59:09.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:59:09.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-14T05:59:11.393 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_46.status 2024-09-14T05:59:11.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:59:11.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:11.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_46 2024-09-14T05:59:11.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_46.status 2024-09-14T05:59:11.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2024-09-14T05:59:11.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:59:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:59:11.452 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=176 2024-09-14T05:59:11.452 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 176 = 176 2024-09-14T05:59:11.452 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:59:11.452 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_46 snap 2024-09-14T05:59:11.452 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:59:11.452 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:11.452 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_46 2024-09-14T05:59:11.452 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:59:11.452 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:59:11.452 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:59:11.452 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:59:12.453 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_46@snap 2024-09-14T05:59:12.502 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_46': 2024-09-14T05:59:12.502 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:59:12.502 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:59:12.502 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-14T05:59:12.502 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c1816305d0 2024-09-14T05:59:12.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c1816305d0 2024-09-14T05:59:12.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:59:12.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:59:12.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:59:12.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:59:12.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:52:08 2024 2024-09-14T05:59:12.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:52:08 2024 2024-09-14T05:59:12.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:52:08 2024 2024-09-14T05:59:12.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:59:12.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:59:12.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:59:12.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 53467864-fd16-454f-8435-46f986b18a2f 2024-09-14T05:59:12.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:59:12.506 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:59:12.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_46 snap 2024-09-14T05:59:12.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:12.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_46 2024-09-14T05:59:12.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:59:12.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:59:12.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_46.export 2024-09-14T05:59:12.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_46.export 2024-09-14T05:59:12.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_46.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_46.export 2024-09-14T05:59:12.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_46@snap /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_46.export 2024-09-14T05:59:12.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:59:12.686 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_46@snap /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_46.export 2024-09-14T05:59:12.915 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:59:12.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_46.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_46.export 2024-09-14T05:59:12.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_46.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_46.export 2024-09-14T05:59:13.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:59:13.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:59:13.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_47 2024-09-14T05:59:13.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_47 snap 2024-09-14T05:59:13.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:59:13.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:13.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_47 2024-09-14T05:59:13.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:59:13.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_47@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T05:59:14.194 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_47 2024-09-14T05:59:14.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:59:14.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:14.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_47 2024-09-14T05:59:14.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_47 started 2024-09-14T05:59:14.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:59:14.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:14.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_47 2024-09-14T05:59:14.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:59:14.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:59:14.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:59:14.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:59:15.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_47 started 2024-09-14T05:59:15.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:59:15.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:15.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_47 2024-09-14T05:59:15.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:59:15.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:59:15.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:59:15.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_47 2024-09-14T05:59:15.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:59:15.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:59:15.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:59:15.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:59:15.198 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:59:15.198 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:59:15.198 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:59:15.198 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:59:15.198 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:59:15.198 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:59:15.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:59:15.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_47 2024-09-14T05:59:15.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:59:15.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:59:15.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:59:15.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:59:15.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:59:15.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:59:15.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:59:15.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:59:15.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:59:15.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:59:15.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:59:15.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:59:15.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:59:15.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:59:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:59:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:59:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:59:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:59:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:59:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:59:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:59:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:59:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:59:15.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:59:15.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:59:15.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:59:15.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:59:15.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:59:15.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:59:15.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:59:15.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:59:15.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:59:15.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:59:15.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_47 2024-09-14T05:59:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-09-14T05:59:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:59:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:59:15.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:59:15.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_47 2024-09-14T05:59:15.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-14T05:59:15.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:59:15.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:59:15.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T05:59:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-14T05:59:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:59:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:59:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:59:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-14T05:59:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T05:59:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:59:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:59:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-14T05:59:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:59:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T05:59:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-14T05:59:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:59:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:59:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T05:59:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-14T05:59:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:59:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:59:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:59:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:59:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T05:59:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:59:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:59:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-14T05:59:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-14T05:59:15.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T05:59:15.327 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:59:15.327 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:59:15.327 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:59:15.327 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_47 2024-09-14T05:59:15.399 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-09-14T05:59:15.399 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:59:15.399 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:59:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:59:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_47 2024-09-14T05:59:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-14T05:59:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:59:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:59:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T05:59:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-14T05:59:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:59:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:59:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:59:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-14T05:59:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T05:59:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:59:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:59:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-14T05:59:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:59:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T05:59:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-14T05:59:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:59:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:59:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T05:59:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-14T05:59:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:59:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:59:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:59:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:59:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T05:59:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:59:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:59:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-14T05:59:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-14T05:59:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T05:59:15.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:59:15.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:59:15.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:59:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_47 2024-09-14T05:59:15.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-09-14T05:59:15.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:59:15.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:59:15.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:59:15.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_47 2024-09-14T05:59:15.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:3 2024-09-14T05:59:15.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:59:15.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:59:15.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-14T05:59:15.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:3 2024-09-14T05:59:15.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:59:15.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:59:15.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:59:15.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=3 2024-09-14T05:59:15.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-14T05:59:15.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:59:15.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:59:15.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=3 2024-09-14T05:59:15.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-14T05:59:15.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-14T05:59:15.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:3 2024-09-14T05:59:15.498 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:59:15.498 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:59:15.498 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-14T05:59:15.498 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:3 2024-09-14T05:59:15.498 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:59:15.498 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:59:15.498 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:59:15.498 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-14T05:59:15.498 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-14T05:59:15.498 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:59:15.498 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:59:15.498 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=3 2024-09-14T05:59:15.498 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=3 2024-09-14T05:59:15.498 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-14T05:59:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T05:59:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T05:59:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T05:59:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_47 2024-09-14T05:59:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:59:15.638 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:59:15.638 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:59:15.638 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:59:15.639 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:59:15.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:59:15.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:59:15.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:59:15.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_47 2024-09-14T05:59:15.640 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:59:15.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:59:15.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:15.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_47 2024-09-14T05:59:15.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:59:15.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:59:15.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_47 2024-09-14T05:59:15.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:59:15.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:59:15.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:15.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_47 2024-09-14T05:59:15.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_47.status 2024-09-14T05:59:15.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_47.status 2024-09-14T05:59:15.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:59:15.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_47.status 2024-09-14T05:59:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_47.status 2024-09-14T05:59:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_47.status 2024-09-14T05:59:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:59:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_47.status 2024-09-14T05:59:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_47 2024-09-14T05:59:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:59:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_47 2024-09-14T05:59:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_47 2024-09-14T05:59:16.190 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 178 2024-09-14T05:59:16.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_47 /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_47.status 2024-09-14T05:59:16.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:59:16.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:16.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_47 2024-09-14T05:59:16.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_47.status 2024-09-14T05:59:16.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_47 --format xml 2024-09-14T05:59:16.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:59:16.250 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:59:16.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=178 2024-09-14T05:59:16.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:59:16.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:59:16.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:59:16.453 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_47.status 2024-09-14T05:59:16.453 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:59:16.453 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:16.453 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_47 2024-09-14T05:59:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_47.status 2024-09-14T05:59:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2024-09-14T05:59:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:59:16.505 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:59:16.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=81 2024-09-14T05:59:16.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 178 = 81 2024-09-14T05:59:16.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:59:16.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:59:16.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_47.status 2024-09-14T05:59:16.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:59:16.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:16.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_47 2024-09-14T05:59:16.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_47.status 2024-09-14T05:59:16.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2024-09-14T05:59:16.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:59:16.960 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:59:16.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=81 2024-09-14T05:59:16.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 178 = 81 2024-09-14T05:59:16.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:59:16.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:59:17.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_47.status 2024-09-14T05:59:17.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:59:17.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:17.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_47 2024-09-14T05:59:17.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_47.status 2024-09-14T05:59:17.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2024-09-14T05:59:17.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:59:17.816 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:59:17.816 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:59:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:59:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:59:17.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:59:17.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:59:17.818 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 178 = '' 2024-09-14T05:59:17.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:59:17.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-14T05:59:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_47.status 2024-09-14T05:59:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:59:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_47 2024-09-14T05:59:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_47.status 2024-09-14T05:59:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2024-09-14T05:59:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:59:19.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:59:19.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=178 2024-09-14T05:59:19.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 178 = 178 2024-09-14T05:59:19.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:59:19.477 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_47 snap 2024-09-14T05:59:19.477 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:59:19.477 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:19.477 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_47 2024-09-14T05:59:19.477 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:59:19.477 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:59:19.477 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:59:19.477 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:59:20.477 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_47@snap 2024-09-14T05:59:20.529 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_47': 2024-09-14T05:59:20.529 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:59:20.529 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:59:20.529 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-14T05:59:20.529 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c1f1f607f8 2024-09-14T05:59:20.529 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c1f1f607f8 2024-09-14T05:59:20.529 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:59:20.529 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:59:20.529 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:59:20.529 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:59:20.529 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:52:23 2024 2024-09-14T05:59:20.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:52:23 2024 2024-09-14T05:59:20.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:52:23 2024 2024-09-14T05:59:20.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:59:20.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:59:20.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:59:20.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 0c24795a-37d5-4c8a-b3a8-c2ecda778c6a 2024-09-14T05:59:20.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:59:20.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:59:20.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_47 snap 2024-09-14T05:59:20.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:20.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_47 2024-09-14T05:59:20.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:59:20.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:59:20.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_47.export 2024-09-14T05:59:20.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_47.export 2024-09-14T05:59:20.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_47.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_47.export 2024-09-14T05:59:20.534 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_47@snap /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_47.export 2024-09-14T05:59:20.697 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:59:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_47@snap /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_47.export 2024-09-14T05:59:20.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:59:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_47.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_47.export 2024-09-14T05:59:20.996 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_47.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_47.export 2024-09-14T05:59:21.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:59:21.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:59:21.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_48 2024-09-14T05:59:21.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_48 snap 2024-09-14T05:59:21.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:59:21.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:21.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_48 2024-09-14T05:59:21.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:59:21.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_48@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T05:59:22.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_48 2024-09-14T05:59:22.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:59:22.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:22.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_48 2024-09-14T05:59:22.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_48 started 2024-09-14T05:59:22.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:59:22.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:22.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_48 2024-09-14T05:59:22.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:59:22.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:59:22.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:59:22.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:59:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_48 started 2024-09-14T05:59:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:59:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_48 2024-09-14T05:59:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:59:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:59:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:59:23.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_48 2024-09-14T05:59:23.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:59:23.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:59:23.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:59:23.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:59:23.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:59:23.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:59:23.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:59:23.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:59:23.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:59:23.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:59:23.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:59:23.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_48 2024-09-14T05:59:23.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:59:23.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:59:23.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:59:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:59:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:59:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:59:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:59:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:59:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:59:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:59:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:59:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:59:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:59:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:59:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:59:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:59:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:59:23.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:59:23.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:59:23.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:59:23.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:59:23.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:59:23.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:59:23.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:59:23.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:59:23.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:59:23.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:59:23.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:59:23.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:59:23.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:59:23.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:59:23.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:59:23.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:59:23.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_48 2024-09-14T05:59:23.312 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2024-09-14T05:59:23.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:59:23.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:59:23.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:59:23.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_48 2024-09-14T05:59:23.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-14T05:59:23.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:59:23.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:59:23.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T05:59:23.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-14T05:59:23.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:59:23.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:59:23.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:59:23.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-14T05:59:23.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T05:59:23.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:59:23.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:59:23.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-14T05:59:23.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:59:23.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T05:59:23.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-14T05:59:23.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:59:23.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:59:23.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T05:59:23.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-14T05:59:23.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:59:23.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:59:23.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:59:23.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:59:23.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T05:59:23.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:59:23.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:59:23.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-14T05:59:23.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-14T05:59:23.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T05:59:23.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:59:23.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:59:23.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:59:23.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_48 2024-09-14T05:59:23.408 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2024-09-14T05:59:23.408 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:59:23.408 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:59:23.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:59:23.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_48 2024-09-14T05:59:23.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-14T05:59:23.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:59:23.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:59:23.411 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T05:59:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-14T05:59:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:59:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:59:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:59:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-14T05:59:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T05:59:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:59:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:59:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-14T05:59:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:59:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T05:59:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-14T05:59:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:59:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:59:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T05:59:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-14T05:59:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:59:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:59:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:59:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:59:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T05:59:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:59:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:59:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-14T05:59:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-14T05:59:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T05:59:23.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:59:23.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:59:23.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:59:23.431 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_48 2024-09-14T05:59:23.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2024-09-14T05:59:23.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:59:23.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:59:23.506 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:59:23.506 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_48 2024-09-14T05:59:23.506 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:3 2024-09-14T05:59:23.506 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:59:23.506 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:59:23.506 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-14T05:59:23.506 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:3 2024-09-14T05:59:23.506 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:59:23.506 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:59:23.506 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:59:23.506 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=3 2024-09-14T05:59:23.506 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-14T05:59:23.506 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:59:23.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:59:23.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=3 2024-09-14T05:59:23.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-14T05:59:23.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-14T05:59:23.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:3 2024-09-14T05:59:23.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:59:23.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:59:23.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-14T05:59:23.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:3 2024-09-14T05:59:23.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:59:23.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:59:23.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:59:23.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-14T05:59:23.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-14T05:59:23.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:59:23.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:59:23.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=3 2024-09-14T05:59:23.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=3 2024-09-14T05:59:23.508 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-14T05:59:23.524 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T05:59:23.524 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T05:59:23.524 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T05:59:23.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_48 2024-09-14T05:59:23.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:59:23.643 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:59:23.643 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:59:23.643 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:59:23.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:59:23.646 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:59:23.646 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:59:23.646 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:59:23.646 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_48 2024-09-14T05:59:23.646 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:59:23.646 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:59:23.646 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:23.646 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_48 2024-09-14T05:59:23.646 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:59:23.646 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:59:23.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_48 2024-09-14T05:59:23.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:59:23.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:59:23.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:23.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_48 2024-09-14T05:59:23.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_48.status 2024-09-14T05:59:23.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_48.status 2024-09-14T05:59:23.647 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:59:23.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_48.status 2024-09-14T05:59:23.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_48.status 2024-09-14T05:59:23.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_48.status 2024-09-14T05:59:23.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:59:23.648 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_48.status 2024-09-14T05:59:23.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_48 2024-09-14T05:59:23.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:59:23.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:23.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_48 2024-09-14T05:59:23.649 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_48 2024-09-14T05:59:24.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 180 2024-09-14T05:59:24.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_48 /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_48.status 2024-09-14T05:59:24.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:59:24.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:24.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_48 2024-09-14T05:59:24.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_48.status 2024-09-14T05:59:24.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_48 --format xml 2024-09-14T05:59:24.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:59:24.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:59:24.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=180 2024-09-14T05:59:24.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:59:24.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:59:24.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:59:24.469 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_48.status 2024-09-14T05:59:24.470 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:59:24.470 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:24.470 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_48 2024-09-14T05:59:24.470 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_48.status 2024-09-14T05:59:24.470 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2024-09-14T05:59:24.470 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:59:24.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:59:24.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=82 2024-09-14T05:59:24.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 180 = 82 2024-09-14T05:59:24.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:59:24.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:59:24.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_48.status 2024-09-14T05:59:24.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:59:24.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:24.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_48 2024-09-14T05:59:24.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_48.status 2024-09-14T05:59:24.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2024-09-14T05:59:24.921 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:59:24.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:59:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=82 2024-09-14T05:59:24.972 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 180 = 82 2024-09-14T05:59:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:59:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:59:25.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_48.status 2024-09-14T05:59:25.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:59:25.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:25.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_48 2024-09-14T05:59:25.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_48.status 2024-09-14T05:59:25.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2024-09-14T05:59:25.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:59:25.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:59:25.824 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:59:25.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:59:25.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:59:25.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:59:25.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:59:25.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 180 = '' 2024-09-14T05:59:25.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:59:25.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-14T05:59:27.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_48.status 2024-09-14T05:59:27.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:59:27.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:27.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_48 2024-09-14T05:59:27.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_48.status 2024-09-14T05:59:27.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2024-09-14T05:59:27.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:59:27.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:59:27.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=180 2024-09-14T05:59:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 180 = 180 2024-09-14T05:59:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:59:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_48 snap 2024-09-14T05:59:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:59:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_48 2024-09-14T05:59:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:59:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:59:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:59:27.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:59:28.486 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_48@snap 2024-09-14T05:59:28.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_48': 2024-09-14T05:59:28.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:59:28.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:59:28.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-14T05:59:28.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c1123b1bc 2024-09-14T05:59:28.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c1123b1bc 2024-09-14T05:59:28.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:59:28.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:59:28.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:59:28.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:59:28.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:52:37 2024 2024-09-14T05:59:28.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:52:37 2024 2024-09-14T05:59:28.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:52:37 2024 2024-09-14T05:59:28.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:59:28.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:59:28.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:59:28.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 630369a5-7829-494e-9cd3-0d8a963a0ae4 2024-09-14T05:59:28.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:59:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:59:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_48 snap 2024-09-14T05:59:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_48 2024-09-14T05:59:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:59:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:59:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_48.export 2024-09-14T05:59:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_48.export 2024-09-14T05:59:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_48.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_48.export 2024-09-14T05:59:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_48@snap /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_48.export 2024-09-14T05:59:28.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:59:28.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_48@snap /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_48.export 2024-09-14T05:59:28.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:59:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_48.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_48.export 2024-09-14T05:59:28.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_48.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_48.export 2024-09-14T05:59:29.018 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:59:29.019 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:59:29.019 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_49 2024-09-14T05:59:29.019 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_49 snap 2024-09-14T05:59:29.019 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:59:29.019 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:29.019 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_49 2024-09-14T05:59:29.019 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:59:29.019 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_49@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T05:59:30.220 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_49 2024-09-14T05:59:30.221 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:59:30.221 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:30.221 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_49 2024-09-14T05:59:30.221 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_49 started 2024-09-14T05:59:30.221 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:59:30.221 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:30.221 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_49 2024-09-14T05:59:30.221 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:59:30.221 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:59:30.221 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:59:30.221 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:59:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_49 started 2024-09-14T05:59:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:59:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_49 2024-09-14T05:59:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:59:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:59:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:59:31.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_49 2024-09-14T05:59:31.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:59:31.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:59:31.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:59:31.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:59:31.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:59:31.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:59:31.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:59:31.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:59:31.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:59:31.225 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:59:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:59:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_49 2024-09-14T05:59:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:59:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:59:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:59:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:59:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:59:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:59:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:59:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:59:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:59:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:59:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:59:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:59:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:59:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:59:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:59:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:59:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:59:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:59:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:59:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:59:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:59:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:59:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:59:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:59:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:59:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:59:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:59:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:59:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:59:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:59:31.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:59:31.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:59:31.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:59:31.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_49 2024-09-14T05:59:31.328 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2024-09-14T05:59:31.328 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:59:31.328 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:59:31.331 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:59:31.331 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_49 2024-09-14T05:59:31.331 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-14T05:59:31.331 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:59:31.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:59:31.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T05:59:31.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-14T05:59:31.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:59:31.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:59:31.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:59:31.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-14T05:59:31.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T05:59:31.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:59:31.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:59:31.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-14T05:59:31.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:59:31.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T05:59:31.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-14T05:59:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:59:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:59:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T05:59:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-14T05:59:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:59:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:59:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:59:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:59:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T05:59:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:59:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:59:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-14T05:59:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-14T05:59:31.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T05:59:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:59:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:59:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:59:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_49 2024-09-14T05:59:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2024-09-14T05:59:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:59:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:59:31.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:59:31.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_49 2024-09-14T05:59:31.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-14T05:59:31.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:59:31.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:59:31.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T05:59:31.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-14T05:59:31.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:59:31.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:59:31.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:59:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-14T05:59:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T05:59:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:59:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:59:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-14T05:59:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:59:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T05:59:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-14T05:59:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:59:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:59:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T05:59:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-14T05:59:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:59:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:59:31.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:59:31.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:59:31.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T05:59:31.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:59:31.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:59:31.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-14T05:59:31.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-14T05:59:31.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T05:59:31.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:59:31.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:59:31.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:59:31.446 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_49 2024-09-14T05:59:31.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2024-09-14T05:59:31.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:59:31.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:59:31.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:59:31.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_49 2024-09-14T05:59:31.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:3 2024-09-14T05:59:31.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:59:31.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:59:31.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-14T05:59:31.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:3 2024-09-14T05:59:31.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:59:31.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:59:31.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:59:31.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=3 2024-09-14T05:59:31.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-14T05:59:31.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:59:31.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:59:31.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=3 2024-09-14T05:59:31.523 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-14T05:59:31.523 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-14T05:59:31.523 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:3 2024-09-14T05:59:31.523 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:59:31.523 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:59:31.523 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-14T05:59:31.523 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:3 2024-09-14T05:59:31.523 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:59:31.523 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:59:31.523 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:59:31.523 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-14T05:59:31.523 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-14T05:59:31.523 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:59:31.523 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:59:31.524 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=3 2024-09-14T05:59:31.524 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=3 2024-09-14T05:59:31.524 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-14T05:59:31.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T05:59:31.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T05:59:31.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T05:59:31.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_49 2024-09-14T05:59:31.658 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:59:31.659 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:59:31.659 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:59:31.659 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:59:31.660 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:59:31.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:59:31.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:59:31.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:59:31.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_49 2024-09-14T05:59:31.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:59:31.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:59:31.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:31.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_49 2024-09-14T05:59:31.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:59:31.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:59:31.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_49 2024-09-14T05:59:31.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:59:31.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:59:31.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:31.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_49 2024-09-14T05:59:31.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_49.status 2024-09-14T05:59:31.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_49.status 2024-09-14T05:59:31.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:59:31.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_49.status 2024-09-14T05:59:31.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_49.status 2024-09-14T05:59:31.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_49.status 2024-09-14T05:59:31.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:59:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_49.status 2024-09-14T05:59:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_49 2024-09-14T05:59:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:59:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_49 2024-09-14T05:59:31.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_49 2024-09-14T05:59:32.220 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 182 2024-09-14T05:59:32.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_49 /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_49.status 2024-09-14T05:59:32.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:59:32.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:32.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_49 2024-09-14T05:59:32.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_49.status 2024-09-14T05:59:32.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_49 --format xml 2024-09-14T05:59:32.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:59:32.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:59:32.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=182 2024-09-14T05:59:32.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:59:32.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:59:32.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:59:32.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_49.status 2024-09-14T05:59:32.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:59:32.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:32.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_49 2024-09-14T05:59:32.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_49.status 2024-09-14T05:59:32.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2024-09-14T05:59:32.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:59:32.532 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:59:32.534 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=83 2024-09-14T05:59:32.534 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 182 = 83 2024-09-14T05:59:32.534 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:59:32.534 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:59:32.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_49.status 2024-09-14T05:59:32.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:59:32.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:32.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_49 2024-09-14T05:59:32.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_49.status 2024-09-14T05:59:32.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2024-09-14T05:59:32.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:59:32.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:59:32.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=83 2024-09-14T05:59:32.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 182 = 83 2024-09-14T05:59:32.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:59:32.986 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:59:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_49.status 2024-09-14T05:59:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:59:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_49 2024-09-14T05:59:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_49.status 2024-09-14T05:59:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2024-09-14T05:59:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:59:33.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:59:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:59:33.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:59:33.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:59:33.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:59:33.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:59:33.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 182 = '' 2024-09-14T05:59:33.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:59:33.842 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-14T05:59:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_49.status 2024-09-14T05:59:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:59:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_49 2024-09-14T05:59:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_49.status 2024-09-14T05:59:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2024-09-14T05:59:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:59:35.498 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:59:35.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=182 2024-09-14T05:59:35.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 182 = 182 2024-09-14T05:59:35.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:59:35.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_49 snap 2024-09-14T05:59:35.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:59:35.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:35.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_49 2024-09-14T05:59:35.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:59:35.502 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:59:35.502 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:59:35.502 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:59:36.502 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_49@snap 2024-09-14T05:59:36.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_49': 2024-09-14T05:59:36.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:59:36.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:59:36.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-14T05:59:36.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c161b86dbf 2024-09-14T05:59:36.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c161b86dbf 2024-09-14T05:59:36.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:59:36.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:59:36.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:59:36.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:59:36.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:52:52 2024 2024-09-14T05:59:36.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:52:52 2024 2024-09-14T05:59:36.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:52:52 2024 2024-09-14T05:59:36.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:59:36.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:59:36.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:59:36.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 2567c386-672b-4b12-ae9d-ec5af9fc0c43 2024-09-14T05:59:36.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:59:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:59:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_49 snap 2024-09-14T05:59:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_49 2024-09-14T05:59:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:59:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:59:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_49.export 2024-09-14T05:59:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_49.export 2024-09-14T05:59:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_49.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_49.export 2024-09-14T05:59:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_49@snap /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_49.export 2024-09-14T05:59:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:59:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_49@snap /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_49.export 2024-09-14T05:59:36.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:59:36.969 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_49.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_49.export 2024-09-14T05:59:37.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_49.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_49.export 2024-09-14T05:59:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:59:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:59:37.055 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_50 2024-09-14T05:59:37.055 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_snap cluster2 mirror image_50 snap 2024-09-14T05:59:37.055 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:59:37.055 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:37.055 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_50 2024-09-14T05:59:37.055 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:59:37.055 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror snap create image_50@snap --debug-rbd=20 --debug-journaler=20 2024-09-14T05:59:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_started cluster1 mirror image_50 2024-09-14T05:59:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:59:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_50 2024-09-14T05:59:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_replay_state cluster1 mirror image_50 started 2024-09-14T05:59:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:59:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_50 2024-09-14T05:59:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=started 2024-09-14T05:59:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:59:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:59:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:59:39.237 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_replay_state cluster1 mirror image_50 started 2024-09-14T05:59:39.237 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:59:39.237 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:39.237 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_50 2024-09-14T05:59:39.237 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=started 2024-09-14T05:59:39.237 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_result 2024-09-14T05:59:39.237 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=stopped 2024-09-14T05:59:39.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_50 2024-09-14T05:59:39.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_instance=cluster1 2024-09-14T05:59:39.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -i state 2024-09-14T05:59:39.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:59:39.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T05:59:39.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance=cluster1 2024-09-14T05:59:39.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local loop_instance 2024-09-14T05:59:39.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-14T05:59:39.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sleep 0 2024-09-14T05:59:39.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-14T05:59:39.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ seq 0 3 2024-09-14T05:59:39.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:59:39.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_50 2024-09-14T05:59:39.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:0 2024-09-14T05:59:39.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:59:39.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:59:39.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-14T05:59:39.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:0 2024-09-14T05:59:39.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:59:39.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:59:39.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:59:39.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T05:59:39.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-14T05:59:39.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:59:39.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:59:39.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=0 2024-09-14T05:59:39.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:59:39.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-14T05:59:39.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:0 2024-09-14T05:59:39.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:59:39.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:59:39.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-14T05:59:39.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:0 2024-09-14T05:59:39.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:59:39.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:59:39.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:59:39.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=0 2024-09-14T05:59:39.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-14T05:59:39.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:59:39.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:59:39.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=0 2024-09-14T05:59:39.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=0 2024-09-14T05:59:39.244 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-14T05:59:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:59:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:59:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-14T05:59:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_50 2024-09-14T05:59:39.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-09-14T05:59:39.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:59:39.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:59:39.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:59:39.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_50 2024-09-14T05:59:39.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:1 2024-09-14T05:59:39.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:59:39.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:59:39.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-14T05:59:39.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:1 2024-09-14T05:59:39.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:59:39.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:59:39.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:59:39.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=1 2024-09-14T05:59:39.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-14T05:59:39.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:59:39.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:59:39.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=1 2024-09-14T05:59:39.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:59:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-14T05:59:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:1 2024-09-14T05:59:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:59:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:59:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-14T05:59:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:1 2024-09-14T05:59:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:59:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:59:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:59:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=1 2024-09-14T05:59:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-14T05:59:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:59:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:59:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=1 2024-09-14T05:59:39.350 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=1 2024-09-14T05:59:39.350 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-14T05:59:39.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:59:39.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:59:39.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-14T05:59:39.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_50 2024-09-14T05:59:39.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-09-14T05:59:39.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:59:39.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:59:39.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:59:39.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_50 2024-09-14T05:59:39.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:2 2024-09-14T05:59:39.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:59:39.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:59:39.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-14T05:59:39.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:2 2024-09-14T05:59:39.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:59:39.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:59:39.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:59:39.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=2 2024-09-14T05:59:39.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-14T05:59:39.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:59:39.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:59:39.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=2 2024-09-14T05:59:39.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:59:39.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-14T05:59:39.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:2 2024-09-14T05:59:39.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:59:39.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:59:39.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-14T05:59:39.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:2 2024-09-14T05:59:39.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:59:39.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:59:39.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:59:39.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=2 2024-09-14T05:59:39.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-14T05:59:39.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:59:39.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:59:39.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=2 2024-09-14T05:59:39.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=2 2024-09-14T05:59:39.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-14T05:59:39.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:59:39.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:59:39.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-14T05:59:39.461 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_50 2024-09-14T05:59:39.532 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-09-14T05:59:39.532 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd mirror status : get status for rbd mirror 2024-09-14T05:59:39.532 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:admin_socket: invalid command 2024-09-14T05:59:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T05:59:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_50 2024-09-14T05:59:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1:3 2024-09-14T05:59:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ shift 2024-09-14T05:59:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance 2024-09-14T05:59:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-14T05:59:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local val=cluster1:3 2024-09-14T05:59:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster_var_name=cluster 2024-09-14T05:59:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local instance_var_name=instance 2024-09-14T05:59:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T05:59:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=3 2024-09-14T05:59:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-14T05:59:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval cluster=cluster1 2024-09-14T05:59:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:59:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ eval instance=3 2024-09-14T05:59:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-14T05:59:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-14T05:59:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local local_cluster=cluster1:3 2024-09-14T05:59:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster=cluster1 2024-09-14T05:59:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance 2024-09-14T05:59:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-14T05:59:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local val=cluster1:3 2024-09-14T05:59:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local cluster_var_name=local_cluster 2024-09-14T05:59:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ local instance_var_name=instance 2024-09-14T05:59:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ cluster=cluster1 2024-09-14T05:59:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ instance=3 2024-09-14T05:59:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-14T05:59:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval local_cluster=cluster1 2024-09-14T05:59:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ local_cluster=cluster1 2024-09-14T05:59:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ eval instance=3 2024-09-14T05:59:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ instance=3 2024-09-14T05:59:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-14T05:59:39.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T05:59:39.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T05:59:39.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-14T05:59:39.555 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_50 2024-09-14T05:59:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ return 0 2024-09-14T05:59:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ status_result=' "state": "Replaying"' 2024-09-14T05:59:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo ' "state": "Replaying"' 2024-09-14T05:59:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ grep -i Replaying 2024-09-14T05:59:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: "state": "Replaying" 2024-09-14T05:59:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=started 2024-09-14T05:59:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test started = started 2024-09-14T05:59:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:59:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_50 2024-09-14T05:59:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:59:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:59:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_50 2024-09-14T05:59:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = journal ']' 2024-09-14T05:59:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' snapshot = snapshot ']' 2024-09-14T05:59:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_50 2024-09-14T05:59:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_cluster=cluster1 2024-09-14T05:59:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:59:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_50 2024-09-14T05:59:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster2-mirror-image_50.status 2024-09-14T05:59:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster2-mirror-image_50.status 2024-09-14T05:59:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:59:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local status_log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_50.status 2024-09-14T05:59:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ mkfname cluster1-mirror-image_50.status 2024-09-14T05:59:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ echo cluster1-mirror-image_50.status 2024-09-14T05:59:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-14T05:59:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local local_status_log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_50.status 2024-09-14T05:59:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ mirror_image_snapshot cluster2 mirror image_50 2024-09-14T05:59:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:59:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_50 2024-09-14T05:59:39.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_50 2024-09-14T05:59:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Snapshot ID: 184 2024-09-14T05:59:40.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_50 /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_50.status 2024-09-14T05:59:40.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:59:40.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:40.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_50 2024-09-14T05:59:40.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_50.status 2024-09-14T05:59:40.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_50 --format xml 2024-09-14T05:59:40.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:59:40.296 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-14T05:59:40.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snapshot_id=184 2024-09-14T05:59:40.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:59:40.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:59:40.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.2 2024-09-14T05:59:40.499 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_50.status 2024-09-14T05:59:40.500 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:59:40.500 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:40.500 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_50 2024-09-14T05:59:40.500 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_50.status 2024-09-14T05:59:40.500 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-09-14T05:59:40.500 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:59:40.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:59:40.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=84 2024-09-14T05:59:40.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 184 = 84 2024-09-14T05:59:40.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:59:40.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.4 2024-09-14T05:59:40.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_50.status 2024-09-14T05:59:40.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:59:40.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:40.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_50 2024-09-14T05:59:40.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_50.status 2024-09-14T05:59:40.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-09-14T05:59:40.955 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:59:41.005 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:59:41.006 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=84 2024-09-14T05:59:41.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 184 = 84 2024-09-14T05:59:41.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:59:41.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.8 2024-09-14T05:59:41.808 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_50.status 2024-09-14T05:59:41.808 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:59:41.808 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:41.808 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_50 2024-09-14T05:59:41.808 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_50.status 2024-09-14T05:59:41.808 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-09-14T05:59:41.808 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:59:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T05:59:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:59:41.865 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:-:1.1: Document is empty 2024-09-14T05:59:41.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: 2024-09-14T05:59:41.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:^ 2024-09-14T05:59:41.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id= 2024-09-14T05:59:41.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 184 = '' 2024-09-14T05:59:41.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-14T05:59:41.866 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1.6 2024-09-14T05:59:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_50.status 2024-09-14T05:59:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:59:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:43.468 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_50 2024-09-14T05:59:43.468 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local log=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_50.status 2024-09-14T05:59:43.468 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-09-14T05:59:43.468 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-14T05:59:43.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-14T05:59:43.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local primary_snapshot_id=184 2024-09-14T05:59:43.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 184 = 184 2024-09-14T05:59:43.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:59:43.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_snap_present cluster1 mirror image_50 snap 2024-09-14T05:59:43.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T05:59:43.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:43.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_50 2024-09-14T05:59:43.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:59:43.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T05:59:43.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-14T05:59:43.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T05:59:44.525 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 info mirror/image_50@snap 2024-09-14T05:59:44.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_50': 2024-09-14T05:59:44.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 128 MiB in 32 objects 2024-09-14T05:59:44.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T05:59:44.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 2 2024-09-14T05:59:44.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 10c1ad7b6824 2024-09-14T05:59:44.581 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.10c1ad7b6824 2024-09-14T05:59:44.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T05:59:44.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff, non-primary 2024-09-14T05:59:44.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T05:59:44.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T05:59:44.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 05:53:00 2024 2024-09-14T05:59:44.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 05:53:00 2024 2024-09-14T05:59:44.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 05:53:00 2024 2024-09-14T05:59:44.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: protected: False 2024-09-14T05:59:44.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T05:59:44.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T05:59:44.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: c89c328d-a242-48de-b9a9-130c74b3f877 2024-09-14T05:59:44.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: false 2024-09-14T05:59:44.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:59:44.585 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ compare_image_snaps mirror image_50 snap 2024-09-14T05:59:44.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:44.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_50 2024-09-14T05:59:44.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap_name=snap 2024-09-14T05:59:44.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local ret=0 2024-09-14T05:59:44.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local rmt_export=/tmp/tmp.sXArEbIrwv/cluster2-mirror-image_50.export 2024-09-14T05:59:44.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local loc_export=/tmp/tmp.sXArEbIrwv/cluster1-mirror-image_50.export 2024-09-14T05:59:44.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_50.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_50.export 2024-09-14T05:59:44.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 -p mirror export image_50@snap /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_50.export 2024-09-14T05:59:44.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:59:44.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster1 -p mirror export image_50@snap /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_50.export 2024-09-14T05:59:45.190 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-14T05:59:45.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cmp /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_50.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_50.export 2024-09-14T05:59:45.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -f /tmp/tmp.sXArEbIrwv/cluster2-mirror-image_50.export /tmp/tmp.sXArEbIrwv/cluster1-mirror-image_50.export 2024-09-14T05:59:45.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:59:45.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ testlog 'TEST: delete many images' 2024-09-14T05:59:45.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ tee -a /tmp/tmp.sXArEbIrwv/rbd-mirror.test.log 2024-09-14T05:59:45.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ date '+%F %T' 2024-09-14T05:59:45.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo 2024-09-14 05:59:45 TEST: delete many images 2024-09-14T05:59:45.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:2024-09-14 05:59:45 TEST: delete many images 2024-09-14T05:59:45.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ seq 1 50 2024-09-14T05:59:45.286 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:59:45.286 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_1 2024-09-14T05:59:45.286 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_1 snap 2024-09-14T05:59:45.286 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:59:45.286 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:45.286 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-14T05:59:45.286 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-14T05:59:45.286 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_1@snap 2024-09-14T05:59:46.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T05:59:46.262 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_1 2024-09-14T05:59:46.262 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:59:46.262 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:46.262 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-14T05:59:46.262 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T05:59:46.262 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T05:59:46.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_1 2024-09-14T05:59:46.264 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:59:46.264 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:46.264 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-14T05:59:46.264 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-09-14T05:59:46.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-09-14T05:59:46.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T05:59:46.434+0000 7f1bc4aa5640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55710cca5ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T05:59:46.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T05:59:46.434+0000 7f1bcb2b2640 -1 rbd::mirror::ImageReplayer: 0x55710d28ef00 [3/cc05da34-933d-4ab7-a8cd-8e584cdd7250] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T05:59:48.402 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T05:59:48.406 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:59:48.406 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:59:48.406 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_2 2024-09-14T05:59:48.406 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_2 snap 2024-09-14T05:59:48.406 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:59:48.406 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:48.407 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-14T05:59:48.407 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-14T05:59:48.407 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_2@snap 2024-09-14T05:59:49.291 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T05:59:49.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_2 2024-09-14T05:59:49.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:59:49.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:49.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-14T05:59:49.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T05:59:49.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T05:59:49.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_2 2024-09-14T05:59:49.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:59:49.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:49.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-14T05:59:49.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-09-14T05:59:49.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-09-14T05:59:49.482 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T05:59:49.481+0000 7f1bc52a6640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55710d38dba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T05:59:49.482 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T05:59:49.481+0000 7f1bcbab3640 -1 rbd::mirror::ImageReplayer: 0x55710d28e000 [3/a89e8edb-c81f-4b3c-b444-29c2fea9908b] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T05:59:51.415 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T05:59:51.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:59:51.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:59:51.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_3 2024-09-14T05:59:51.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_3 snap 2024-09-14T05:59:51.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:59:51.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:51.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-14T05:59:51.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-14T05:59:51.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_3@snap 2024-09-14T05:59:52.293 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T05:59:52.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_3 2024-09-14T05:59:52.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:59:52.322 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:52.322 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-14T05:59:52.322 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T05:59:52.322 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T05:59:52.323 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_3 2024-09-14T05:59:52.323 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:59:52.323 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:52.323 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-14T05:59:52.323 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-09-14T05:59:52.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-09-14T05:59:52.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T05:59:52.490+0000 7f1bc52a6640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55710d38da00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T05:59:52.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T05:59:52.490+0000 7f1bcb2b2640 -1 rbd::mirror::ImageReplayer: 0x55710d33e280 [3/847da547-970e-41a0-bee0-110873986de0] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T05:59:54.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T05:59:54.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:59:54.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:59:54.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_4 2024-09-14T05:59:54.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_4 snap 2024-09-14T05:59:54.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:59:54.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:54.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-14T05:59:54.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-14T05:59:54.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_4@snap 2024-09-14T05:59:55.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T05:59:55.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_4 2024-09-14T05:59:55.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:59:55.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:55.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-14T05:59:55.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T05:59:55.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T05:59:55.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_4 2024-09-14T05:59:55.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:59:55.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:55.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-14T05:59:55.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-09-14T05:59:55.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-09-14T05:59:55.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T05:59:55.510+0000 7f1bc52a6640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55710cca5d40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T05:59:55.512 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T05:59:55.510+0000 7f1bcb2b2640 -1 rbd::mirror::ImageReplayer: 0x55710d361400 [3/476c0897-664e-4ed7-a165-44f4a9d2c1a5] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T05:59:57.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T05:59:57.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T05:59:57.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T05:59:57.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_5 2024-09-14T05:59:57.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_5 snap 2024-09-14T05:59:57.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:59:57.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:57.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-14T05:59:57.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-14T05:59:57.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_5@snap 2024-09-14T05:59:58.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T05:59:58.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_5 2024-09-14T05:59:58.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:59:58.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:58.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-14T05:59:58.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T05:59:58.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T05:59:58.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_5 2024-09-14T05:59:58.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T05:59:58.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T05:59:58.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-14T05:59:58.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-09-14T05:59:58.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-09-14T05:59:58.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T05:59:58.322+0000 7f1bc4aa5640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55710c7971e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T05:59:58.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T05:59:58.322+0000 7f1bcb2b2640 -1 rbd::mirror::ImageReplayer: 0x55710d308280 [3/5962be87-a0c9-4726-93e1-f3aad0bd9173] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T06:00:00.451 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:00:00.455 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:00:00.455 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:00:00.455 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_6 2024-09-14T06:00:00.455 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_6 snap 2024-09-14T06:00:00.455 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:00.455 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:00.455 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-14T06:00:00.455 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-14T06:00:00.455 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_6@snap 2024-09-14T06:00:00.525 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9179045843660958) > weight (0.25) 2024-09-14T06:00:00.525 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5298687265217719) > weight (0.25) 2024-09-14T06:00:00.525 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-09-14T06:00:00.526 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-09-14T06:00:00.526 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-09-14T06:00:00.526 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-09-14T06:00:00.526 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T06:00:00.527 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-14T06:00:00.526+0000 7f1a0be04640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 59618) UID: 1000 2024-09-14T06:00:01.331 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T06:00:01.352 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_6 2024-09-14T06:00:01.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:01.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:01.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-14T06:00:01.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:00:01.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:00:01.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_6 2024-09-14T06:00:01.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:01.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:01.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-14T06:00:01.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-09-14T06:00:01.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-09-14T06:00:01.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:00:01.520+0000 7f1bc4aa5640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55710cca5040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T06:00:01.522 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:00:01.520+0000 7f1bcb2b2640 -1 rbd::mirror::ImageReplayer: 0x55710d33fb80 [3/a86da690-7adb-435f-a4fb-fb58a8d79cb7] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T06:00:03.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:00:03.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:00:03.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:00:03.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_7 2024-09-14T06:00:03.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_7 snap 2024-09-14T06:00:03.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:03.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:03.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-14T06:00:03.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-14T06:00:03.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_7@snap 2024-09-14T06:00:03.717 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-14T06:00:03.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-14T06:00:03.717 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-09-14T06:00:03.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-14T06:00:03.718+0000 7f91210e8640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 59512) UID: 1000 2024-09-14T06:00:04.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T06:00:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_7 2024-09-14T06:00:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-14T06:00:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:00:04.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:00:04.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_7 2024-09-14T06:00:04.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:04.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:04.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-14T06:00:04.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-09-14T06:00:04.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-09-14T06:00:04.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:00:04.314+0000 7f1bc4aa5640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55710cca51e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T06:00:04.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:00:04.314+0000 7f1bcb2b2640 -1 rbd::mirror::ImageReplayer: 0x55710d360500 [3/34d473d8-f5ad-4be8-be0a-057254b091f7] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T06:00:04.718 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-14T06:00:04.718 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T06:00:04.718 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-14T06:00:04.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-14T06:00:04.719 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-14T06:00:04.722 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-14T06:00:04.722 INFO:tasks.rbd_mirror_thrash:waiting for 105 secs before thrashing 2024-09-14T06:00:06.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:00:06.264 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:00:06.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:00:06.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_8 2024-09-14T06:00:06.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_8 snap 2024-09-14T06:00:06.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:06.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:06.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-14T06:00:06.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-14T06:00:06.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_8@snap 2024-09-14T06:00:06.628 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-09-14T06:00:06.629 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-14T06:00:06.629 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-14T06:00:06.632 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-14T06:00:06.632 INFO:tasks.rbd_mirror_thrash:waiting for 103 secs before thrashing 2024-09-14T06:00:07.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T06:00:07.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_8 2024-09-14T06:00:07.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:07.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:07.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-14T06:00:07.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:00:07.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:00:07.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_8 2024-09-14T06:00:07.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:07.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:07.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-14T06:00:07.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2024-09-14T06:00:07.203 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2024-09-14T06:00:07.314 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:00:07.313+0000 7f1bc52a6640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55710cca5380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T06:00:07.314 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:00:07.313+0000 7f1bcbab3640 -1 rbd::mirror::ImageReplayer: 0x55710cdb9900 [3/21958cb0-e88b-48db-bd17-5fa7463f50e0] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T06:00:09.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:00:09.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:00:09.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:00:09.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_9 2024-09-14T06:00:09.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_9 snap 2024-09-14T06:00:09.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:09.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:09.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-14T06:00:09.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-14T06:00:09.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_9@snap 2024-09-14T06:00:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T06:00:10.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_9 2024-09-14T06:00:10.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:10.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:10.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-14T06:00:10.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:00:10.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:00:10.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_9 2024-09-14T06:00:10.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:10.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:10.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-14T06:00:10.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2024-09-14T06:00:10.228 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2024-09-14T06:00:10.343 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:00:10.343+0000 7f1bc52a6640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55710d38d520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T06:00:10.343 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:00:10.343+0000 7f1bcbab3640 -1 rbd::mirror::ImageReplayer: 0x55710d361900 [3/8d3c3903-631c-42c7-8d30-a5b307b883eb] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T06:00:12.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:00:12.286 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:00:12.286 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:00:12.286 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_10 2024-09-14T06:00:12.286 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_10 snap 2024-09-14T06:00:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-14T06:00:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-14T06:00:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_10@snap 2024-09-14T06:00:13.128 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T06:00:13.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_10 2024-09-14T06:00:13.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:13.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:13.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-14T06:00:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:00:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:00:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_10 2024-09-14T06:00:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-14T06:00:13.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2024-09-14T06:00:13.216 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2024-09-14T06:00:13.325 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:00:13.324+0000 7f1bc52a6640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55710c4fb1e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T06:00:13.325 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:00:13.324+0000 7f1bcb2b2640 -1 rbd::mirror::ImageReplayer: 0x55710d28ea00 [3/c0427ec2-f759-4269-ac43-364d5f6678fa] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T06:00:15.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:00:15.287 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:00:15.288 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:00:15.288 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_11 2024-09-14T06:00:15.288 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_11 snap 2024-09-14T06:00:15.288 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:15.288 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:15.288 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_11 2024-09-14T06:00:15.288 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-14T06:00:15.288 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_11@snap 2024-09-14T06:00:16.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T06:00:16.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_11 2024-09-14T06:00:16.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:16.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:16.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_11 2024-09-14T06:00:16.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:00:16.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:00:16.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_11 2024-09-14T06:00:16.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:16.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:16.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_11 2024-09-14T06:00:16.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_11 2024-09-14T06:00:16.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_11 2024-09-14T06:00:16.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:00:16.360+0000 7f1bc52a6640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55710c4fa340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T06:00:16.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:00:16.360+0000 7f1bcb2b2640 -1 rbd::mirror::ImageReplayer: 0x55710d28e500 [3/b6cea04e-7839-4e62-8e00-34e757bf5550] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T06:00:18.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:00:18.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:00:18.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:00:18.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_12 2024-09-14T06:00:18.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_12 snap 2024-09-14T06:00:18.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:18.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:18.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_12 2024-09-14T06:00:18.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-14T06:00:18.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_12@snap 2024-09-14T06:00:19.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T06:00:19.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_12 2024-09-14T06:00:19.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:19.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:19.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_12 2024-09-14T06:00:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:00:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:00:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_12 2024-09-14T06:00:19.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:19.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:19.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_12 2024-09-14T06:00:19.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_12 2024-09-14T06:00:19.262 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_12 2024-09-14T06:00:19.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:00:19.377+0000 7f1bc52a6640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55710c4fa680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T06:00:19.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:00:19.377+0000 7f1bcb2b2640 -1 rbd::mirror::ImageReplayer: 0x55710d309b80 [3/7b610a90-6bb9-4ffb-8270-bbf747673d2a] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T06:00:21.312 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:00:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:00:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:00:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_13 2024-09-14T06:00:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_13 snap 2024-09-14T06:00:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_13 2024-09-14T06:00:21.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-14T06:00:21.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_13@snap 2024-09-14T06:00:22.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T06:00:22.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_13 2024-09-14T06:00:22.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:22.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:22.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_13 2024-09-14T06:00:22.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:00:22.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:00:22.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_13 2024-09-14T06:00:22.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:22.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:22.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_13 2024-09-14T06:00:22.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_13 2024-09-14T06:00:22.270 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_13 2024-09-14T06:00:22.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:00:22.378+0000 7f1bc52a6640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55710c877380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T06:00:22.380 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:00:22.378+0000 7f1bcbab3640 -1 rbd::mirror::ImageReplayer: 0x55710cdb8780 [3/03d4b3e9-bc59-4da0-9ad0-74a331dde7bb] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T06:00:24.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:00:24.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:00:24.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:00:24.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_14 2024-09-14T06:00:24.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_14 snap 2024-09-14T06:00:24.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:24.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:24.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_14 2024-09-14T06:00:24.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-14T06:00:24.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_14@snap 2024-09-14T06:00:25.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T06:00:25.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_14 2024-09-14T06:00:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_14 2024-09-14T06:00:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:00:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:00:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_14 2024-09-14T06:00:25.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:25.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:25.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_14 2024-09-14T06:00:25.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_14 2024-09-14T06:00:25.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_14 2024-09-14T06:00:25.394 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:00:25.392+0000 7f1bc4aa5640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55710cca4680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T06:00:25.394 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:00:25.392+0000 7f1bcb2b2640 -1 rbd::mirror::ImageReplayer: 0x55710d309900 [3/72651fba-3b42-454d-a354-6f5b7be1dd8d] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T06:00:27.330 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:00:27.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:00:27.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:00:27.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_15 2024-09-14T06:00:27.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_15 snap 2024-09-14T06:00:27.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:27.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:27.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_15 2024-09-14T06:00:27.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-14T06:00:27.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_15@snap 2024-09-14T06:00:28.141 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T06:00:28.160 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_15 2024-09-14T06:00:28.160 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:28.161 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:28.161 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_15 2024-09-14T06:00:28.161 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:00:28.161 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:00:28.161 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_15 2024-09-14T06:00:28.161 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:28.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:28.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_15 2024-09-14T06:00:28.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_15 2024-09-14T06:00:28.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_15 2024-09-14T06:00:28.329 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:00:28.328+0000 7f1bc4aa5640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55710c877520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T06:00:28.329 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:00:28.328+0000 7f1bcbab3640 -1 rbd::mirror::ImageReplayer: 0x55710d361b80 [3/8e5e2fa0-ccc2-4850-a80c-d8985387aa83] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T06:00:30.340 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:00:30.343 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:00:30.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:00:30.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_16 2024-09-14T06:00:30.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_16 snap 2024-09-14T06:00:30.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:30.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:30.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_16 2024-09-14T06:00:30.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-14T06:00:30.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_16@snap 2024-09-14T06:00:31.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T06:00:31.264 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_16 2024-09-14T06:00:31.264 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:31.264 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:31.264 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_16 2024-09-14T06:00:31.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:00:31.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:00:31.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_16 2024-09-14T06:00:31.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:31.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:31.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_16 2024-09-14T06:00:31.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_16 2024-09-14T06:00:31.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_16 2024-09-14T06:00:31.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:00:31.434+0000 7f1bc4aa5640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55710c8776c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T06:00:31.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:00:31.434+0000 7f1bcbab3640 -1 rbd::mirror::ImageReplayer: 0x55710cdb9680 [3/1fcc0d0b-34ef-46ef-b563-beb55715aefe] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T06:00:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:00:33.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:00:33.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:00:33.262 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_17 2024-09-14T06:00:33.262 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_17 snap 2024-09-14T06:00:33.262 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:33.262 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:33.262 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_17 2024-09-14T06:00:33.262 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-14T06:00:33.262 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_17@snap 2024-09-14T06:00:34.226 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T06:00:34.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_17 2024-09-14T06:00:34.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:34.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:34.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_17 2024-09-14T06:00:34.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:00:34.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:00:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_17 2024-09-14T06:00:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:34.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_17 2024-09-14T06:00:34.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_17 2024-09-14T06:00:34.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_17 2024-09-14T06:00:34.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:00:34.430+0000 7f1bc4aa5640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55710cca56c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T06:00:34.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:00:34.430+0000 7f1bcb2b2640 -1 rbd::mirror::ImageReplayer: 0x55710d308500 [3/5c00a5fc-12ce-46ba-b4c0-0e6aa090fe1a] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T06:00:36.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:00:36.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:00:36.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:00:36.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_18 2024-09-14T06:00:36.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_18 snap 2024-09-14T06:00:36.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:36.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:36.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_18 2024-09-14T06:00:36.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-14T06:00:36.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_18@snap 2024-09-14T06:00:37.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T06:00:37.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_18 2024-09-14T06:00:37.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:37.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:37.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_18 2024-09-14T06:00:37.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:00:37.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:00:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_18 2024-09-14T06:00:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_18 2024-09-14T06:00:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_18 2024-09-14T06:00:37.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_18 2024-09-14T06:00:37.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:00:37.433+0000 7f1bc52a6640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55710c78b1e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T06:00:37.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:00:37.433+0000 7f1bcbab3640 -1 rbd::mirror::ImageReplayer: 0x55710d360c80 [3/3d117385-98ae-4587-b863-f50b36e012f3] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T06:00:39.462 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:00:39.465 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:00:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:00:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_19 2024-09-14T06:00:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_19 snap 2024-09-14T06:00:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_19 2024-09-14T06:00:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-14T06:00:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_19@snap 2024-09-14T06:00:40.133 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T06:00:40.158 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_19 2024-09-14T06:00:40.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:40.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:40.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_19 2024-09-14T06:00:40.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:00:40.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:00:40.160 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_19 2024-09-14T06:00:40.160 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:40.160 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:40.160 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_19 2024-09-14T06:00:40.160 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_19 2024-09-14T06:00:40.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_19 2024-09-14T06:00:40.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:00:40.326+0000 7f1bc52a6640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55710c78b380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T06:00:40.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:00:40.326+0000 7f1bcb2b2640 -1 rbd::mirror::ImageReplayer: 0x55710d33f680 [3/a166af45-2737-4c1c-912c-26014f069f63] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T06:00:42.079 INFO:tasks.ceph.cluster1.osd.0.smithi143.stderr:2024-09-14T06:00:42.078+0000 7f890ab46640 -1 reset not still connected to 0x557d63cd7380 2024-09-14T06:00:42.084 INFO:tasks.ceph.cluster1.osd.0.smithi143.stderr:2024-09-14T06:00:42.083+0000 7f890ab46640 -1 reset not still connected to 0x557d63cd76c0 2024-09-14T06:00:42.264 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:00:42.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:00:42.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:00:42.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_20 2024-09-14T06:00:42.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_20 snap 2024-09-14T06:00:42.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:42.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:42.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_20 2024-09-14T06:00:42.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-14T06:00:42.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_20@snap 2024-09-14T06:00:43.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T06:00:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_20 2024-09-14T06:00:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_20 2024-09-14T06:00:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:00:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:00:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_20 2024-09-14T06:00:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:43.172 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:43.172 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_20 2024-09-14T06:00:43.172 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_20 2024-09-14T06:00:43.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_20 2024-09-14T06:00:43.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:00:43.335+0000 7f1bc4aa5640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55710c78b520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T06:00:43.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:00:43.335+0000 7f1bcb2b2640 -1 rbd::mirror::ImageReplayer: 0x55710d309180 [3/6654bddc-72e9-47b0-8061-615cff98334f] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T06:00:45.273 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:00:45.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:00:45.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:00:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_21 2024-09-14T06:00:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_21 snap 2024-09-14T06:00:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_21 2024-09-14T06:00:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-14T06:00:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_21@snap 2024-09-14T06:00:46.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T06:00:46.187 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_21 2024-09-14T06:00:46.187 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:46.187 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:46.187 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_21 2024-09-14T06:00:46.187 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:00:46.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:00:46.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_21 2024-09-14T06:00:46.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:46.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:46.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_21 2024-09-14T06:00:46.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_21 2024-09-14T06:00:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_21 2024-09-14T06:00:46.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:00:46.357+0000 7f1bc52a6640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55710cca4820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T06:00:46.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:00:46.357+0000 7f1bcb2b2640 -1 rbd::mirror::ImageReplayer: 0x55710d33f400 [3/a03aba9f-8291-447c-bf72-b41385e7c989] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T06:00:48.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:00:48.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:00:48.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:00:48.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_22 2024-09-14T06:00:48.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_22 snap 2024-09-14T06:00:48.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:48.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:48.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_22 2024-09-14T06:00:48.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-14T06:00:48.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_22@snap 2024-09-14T06:00:49.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T06:00:49.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_22 2024-09-14T06:00:49.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:49.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:49.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_22 2024-09-14T06:00:49.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:00:49.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:00:49.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_22 2024-09-14T06:00:49.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:49.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:49.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_22 2024-09-14T06:00:49.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_22 2024-09-14T06:00:49.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_22 2024-09-14T06:00:49.765 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:00:49.764+0000 7f1bc4aa5640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55710d302340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T06:00:49.765 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:00:49.765+0000 7f1bcbab3640 -1 rbd::mirror::ImageReplayer: 0x55710d33e000 [3/81e2e849-43ea-4a3d-825a-ccda8c67058a] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T06:00:51.302 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:00:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:00:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:00:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_23 2024-09-14T06:00:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_23 snap 2024-09-14T06:00:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_23 2024-09-14T06:00:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-14T06:00:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_23@snap 2024-09-14T06:00:52.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T06:00:52.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_23 2024-09-14T06:00:52.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:52.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:52.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_23 2024-09-14T06:00:52.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:00:52.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:00:52.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_23 2024-09-14T06:00:52.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:52.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:52.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_23 2024-09-14T06:00:52.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_23 2024-09-14T06:00:52.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_23 2024-09-14T06:00:52.372 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:00:52.371+0000 7f1bc52a6640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55710d302820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T06:00:52.372 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:00:52.371+0000 7f1bcbab3640 -1 rbd::mirror::ImageReplayer: 0x55710cdb9400 [3/1ad775b2-4acb-4f43-b70c-7d7c1bb4782d] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T06:00:54.262 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:00:54.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:00:54.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:00:54.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_24 2024-09-14T06:00:54.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_24 snap 2024-09-14T06:00:54.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:54.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:54.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_24 2024-09-14T06:00:54.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-14T06:00:54.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_24@snap 2024-09-14T06:00:55.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T06:00:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_24 2024-09-14T06:00:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_24 2024-09-14T06:00:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:00:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:00:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_24 2024-09-14T06:00:55.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:55.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:55.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_24 2024-09-14T06:00:55.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_24 2024-09-14T06:00:55.271 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_24 2024-09-14T06:00:55.581 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:00:55.580+0000 7f1bc4aa5640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55710d302ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T06:00:55.581 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:00:55.580+0000 7f1bcb2b2640 -1 rbd::mirror::ImageReplayer: 0x55710cdb8a00 [3/0470f29c-7ca8-419d-8a53-5b2f3f342a4f] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T06:00:57.111 INFO:tasks.ceph.cluster1.osd.2.smithi143.stderr:2024-09-14T06:00:57.110+0000 7ff5a47b4640 -1 reset not still connected to 0x55c40d3ba0d0 2024-09-14T06:00:57.111 INFO:tasks.ceph.cluster1.osd.2.smithi143.stderr:2024-09-14T06:00:57.110+0000 7ff5a47b4640 -1 reset not still connected to 0x55c40d3ba4e0 2024-09-14T06:00:57.111 INFO:tasks.ceph.cluster1.osd.2.smithi143.stderr:2024-09-14T06:00:57.110+0000 7ff5a47b4640 -1 reset not still connected to 0x55c40d3ba5b0 2024-09-14T06:00:57.111 INFO:tasks.ceph.cluster1.osd.2.smithi143.stderr:2024-09-14T06:00:57.110+0000 7ff5a47b4640 -1 reset not still connected to 0x55c40d3ba750 2024-09-14T06:00:57.111 INFO:tasks.ceph.cluster1.osd.2.smithi143.stderr:2024-09-14T06:00:57.110+0000 7ff5a47b4640 -1 reset not still connected to 0x55c40d3baa90 2024-09-14T06:00:57.111 INFO:tasks.ceph.cluster1.osd.2.smithi143.stderr:2024-09-14T06:00:57.110+0000 7ff5a47b4640 -1 reset not still connected to 0x55c40dd16a90 2024-09-14T06:00:57.111 INFO:tasks.ceph.cluster1.osd.2.smithi143.stderr:2024-09-14T06:00:57.110+0000 7ff5a47b4640 -1 reset not still connected to 0x55c40de48680 2024-09-14T06:00:57.111 INFO:tasks.ceph.cluster1.osd.2.smithi143.stderr:2024-09-14T06:00:57.110+0000 7ff5a47b4640 -1 reset not still connected to 0x55c40df94340 2024-09-14T06:00:57.112 INFO:tasks.ceph.cluster1.osd.2.smithi143.stderr:2024-09-14T06:00:57.110+0000 7ff5a47b4640 -1 reset not still connected to 0x55c40df94d00 2024-09-14T06:00:57.112 INFO:tasks.ceph.cluster1.osd.2.smithi143.stderr:2024-09-14T06:00:57.111+0000 7ff5a47b4640 -1 reset not still connected to 0x55c40df95ba0 2024-09-14T06:00:57.112 INFO:tasks.ceph.cluster1.osd.2.smithi143.stderr:2024-09-14T06:00:57.111+0000 7ff5a47b4640 -1 reset not still connected to 0x55c40e5648f0 2024-09-14T06:00:57.113 INFO:tasks.ceph.cluster1.osd.2.smithi143.stderr:2024-09-14T06:00:57.111+0000 7ff5a47b4640 -1 reset not still connected to 0x55c40e564c30 2024-09-14T06:00:57.113 INFO:tasks.ceph.cluster1.osd.2.smithi143.stderr:2024-09-14T06:00:57.111+0000 7ff5a47b4640 -1 reset not still connected to 0x55c412fde0d0 2024-09-14T06:00:57.143 INFO:tasks.ceph.cluster1.osd.0.smithi143.stderr:2024-09-14T06:00:57.142+0000 7f890ab46640 -1 reset not still connected to 0x557d6072e0d0 2024-09-14T06:00:57.143 INFO:tasks.ceph.cluster1.osd.0.smithi143.stderr:2024-09-14T06:00:57.142+0000 7f890ab46640 -1 reset not still connected to 0x557d6072e1a0 2024-09-14T06:00:57.143 INFO:tasks.ceph.cluster1.osd.0.smithi143.stderr:2024-09-14T06:00:57.142+0000 7f890ab46640 -1 reset not still connected to 0x557d6072e340 2024-09-14T06:00:57.143 INFO:tasks.ceph.cluster1.osd.0.smithi143.stderr:2024-09-14T06:00:57.142+0000 7f890ab46640 -1 reset not still connected to 0x557d63cd68f0 2024-09-14T06:00:57.144 INFO:tasks.ceph.cluster1.osd.0.smithi143.stderr:2024-09-14T06:00:57.142+0000 7f890ab46640 -1 reset not still connected to 0x557d63cd6a90 2024-09-14T06:00:57.146 INFO:tasks.ceph.cluster1.osd.0.smithi143.stderr:2024-09-14T06:00:57.145+0000 7f890ab46640 -1 reset not still connected to 0x557d63cd7380 2024-09-14T06:00:57.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:00:57.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:00:57.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:00:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_25 2024-09-14T06:00:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_25 snap 2024-09-14T06:00:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_25 2024-09-14T06:00:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-14T06:00:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_25@snap 2024-09-14T06:00:58.134 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T06:00:58.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_25 2024-09-14T06:00:58.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:58.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:58.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_25 2024-09-14T06:00:58.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:00:58.164 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:00:58.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_25 2024-09-14T06:00:58.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:00:58.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:00:58.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_25 2024-09-14T06:00:58.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_25 2024-09-14T06:00:58.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_25 2024-09-14T06:00:58.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:00:58.537+0000 7f1bc52a6640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55710d302b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T06:00:58.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:00:58.537+0000 7f1bcbab3640 -1 rbd::mirror::ImageReplayer: 0x55710d33f900 [3/a6b55bc2-53d1-494f-9e17-76231088f41c] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T06:01:00.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:01:00.340 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:01:00.340 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:01:00.340 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_26 2024-09-14T06:01:00.340 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_26 snap 2024-09-14T06:01:00.340 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:00.340 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:00.341 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_26 2024-09-14T06:01:00.341 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-14T06:01:00.341 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_26@snap 2024-09-14T06:01:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T06:01:01.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_26 2024-09-14T06:01:01.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:01.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:01.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_26 2024-09-14T06:01:01.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:01:01.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:01:01.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_26 2024-09-14T06:01:01.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:01.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:01.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_26 2024-09-14T06:01:01.240 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_26 2024-09-14T06:01:01.292 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_26 2024-09-14T06:01:01.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:01:01.409+0000 7f1bc4aa5640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55710c78b6c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T06:01:01.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:01:01.409+0000 7f1bcb2b2640 -1 rbd::mirror::ImageReplayer: 0x55710d33f180 [3/9154d447-2cac-4f96-97e0-0ce2dbafe588] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T06:01:03.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:01:03.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:01:03.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:01:03.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_27 2024-09-14T06:01:03.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_27 snap 2024-09-14T06:01:03.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:03.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:03.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_27 2024-09-14T06:01:03.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-14T06:01:03.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_27@snap 2024-09-14T06:01:04.226 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T06:01:04.238 INFO:tasks.ceph.cluster2.osd.0.smithi191.stderr:2024-09-14T06:01:04.231+0000 7f75dca16640 -1 reset not still connected to 0x55ea91b3f1e0 2024-09-14T06:01:04.248 INFO:tasks.ceph.cluster2.osd.0.smithi191.stderr:2024-09-14T06:01:04.247+0000 7f75dca16640 -1 reset not still connected to 0x55ea92acf520 2024-09-14T06:01:04.248 INFO:tasks.ceph.cluster2.osd.0.smithi191.stderr:2024-09-14T06:01:04.247+0000 7f75dca16640 -1 reset not still connected to 0x55ea92acf5f0 2024-09-14T06:01:04.248 INFO:tasks.ceph.cluster2.osd.0.smithi191.stderr:2024-09-14T06:01:04.247+0000 7f75dca16640 -1 reset not still connected to 0x55ea92acfa00 2024-09-14T06:01:04.248 INFO:tasks.ceph.cluster2.osd.0.smithi191.stderr:2024-09-14T06:01:04.247+0000 7f75dca16640 -1 reset not still connected to 0x55ea9e844dd0 2024-09-14T06:01:04.248 INFO:tasks.ceph.cluster2.osd.0.smithi191.stderr:2024-09-14T06:01:04.247+0000 7f75dca16640 -1 reset not still connected to 0x55eaa1ef8820 2024-09-14T06:01:04.248 INFO:tasks.ceph.cluster2.osd.0.smithi191.stderr:2024-09-14T06:01:04.247+0000 7f75dca16640 -1 reset not still connected to 0x55eaa1ef96c0 2024-09-14T06:01:04.248 INFO:tasks.ceph.cluster2.osd.0.smithi191.stderr:2024-09-14T06:01:04.247+0000 7f75dca16640 -1 reset not still connected to 0x55eaa1ef9ba0 2024-09-14T06:01:04.248 INFO:tasks.ceph.cluster2.osd.0.smithi191.stderr:2024-09-14T06:01:04.247+0000 7f75dca16640 -1 reset not still connected to 0x55eaa59e7040 2024-09-14T06:01:04.248 INFO:tasks.ceph.cluster2.osd.0.smithi191.stderr:2024-09-14T06:01:04.247+0000 7f75dca16640 -1 reset not still connected to 0x55eaa59e7a00 2024-09-14T06:01:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_27 2024-09-14T06:01:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:04.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_27 2024-09-14T06:01:04.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:01:04.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:01:04.262 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_27 2024-09-14T06:01:04.262 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:04.262 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:04.262 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_27 2024-09-14T06:01:04.262 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_27 2024-09-14T06:01:04.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_27 2024-09-14T06:01:04.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:01:04.427+0000 7f1bc52a6640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55710c78b860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T06:01:04.429 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:01:04.427+0000 7f1bcb2b2640 -1 rbd::mirror::ImageReplayer: 0x55710d28ec80 [3/c4de874f-fa8c-4df7-a048-8394cb50d034] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T06:01:06.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:01:06.371 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:01:06.371 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:01:06.371 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_28 2024-09-14T06:01:06.371 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_28 snap 2024-09-14T06:01:06.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:06.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:06.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_28 2024-09-14T06:01:06.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-14T06:01:06.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_28@snap 2024-09-14T06:01:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T06:01:07.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_28 2024-09-14T06:01:07.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:07.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:07.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_28 2024-09-14T06:01:07.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:01:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:01:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_28 2024-09-14T06:01:07.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:07.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:07.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_28 2024-09-14T06:01:07.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_28 2024-09-14T06:01:07.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_28 2024-09-14T06:01:07.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:01:07.450+0000 7f1bc4aa5640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55710c78ba00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T06:01:07.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:01:07.450+0000 7f1bcb2b2640 -1 rbd::mirror::ImageReplayer: 0x55710d308780 [3/5c71f4af-d4cf-4db0-96fd-a68b7ed67897] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T06:01:09.271 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:01:09.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:01:09.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:01:09.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_29 2024-09-14T06:01:09.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_29 snap 2024-09-14T06:01:09.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:09.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:09.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_29 2024-09-14T06:01:09.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-14T06:01:09.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_29@snap 2024-09-14T06:01:10.160 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T06:01:10.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_29 2024-09-14T06:01:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_29 2024-09-14T06:01:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:01:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:01:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_29 2024-09-14T06:01:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_29 2024-09-14T06:01:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_29 2024-09-14T06:01:10.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_29 2024-09-14T06:01:10.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:01:10.346+0000 7f1bc52a6640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55710c78bba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T06:01:10.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:01:10.346+0000 7f1bcbab3640 -1 rbd::mirror::ImageReplayer: 0x55710d360780 [3/3672d66c-7b41-4af2-9094-129bf558bdc1] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T06:01:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:01:12.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:01:12.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:01:12.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_30 2024-09-14T06:01:12.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_30 snap 2024-09-14T06:01:12.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:12.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:12.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_30 2024-09-14T06:01:12.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-14T06:01:12.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_30@snap 2024-09-14T06:01:13.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T06:01:13.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_30 2024-09-14T06:01:13.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:13.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_30 2024-09-14T06:01:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:01:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:01:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_30 2024-09-14T06:01:13.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:13.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:13.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_30 2024-09-14T06:01:13.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_30 2024-09-14T06:01:13.227 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_30 2024-09-14T06:01:13.341 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:01:13.340+0000 7f1bc4aa5640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55710d303380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T06:01:13.341 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:01:13.340+0000 7f1bcb2b2640 -1 rbd::mirror::ImageReplayer: 0x55710d360f00 [3/3f054947-dd92-44d7-bb13-91eab274073e] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T06:01:15.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:01:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:01:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:01:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_31 2024-09-14T06:01:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_31 snap 2024-09-14T06:01:15.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:15.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:15.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_31 2024-09-14T06:01:15.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-14T06:01:15.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_31@snap 2024-09-14T06:01:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T06:01:16.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_31 2024-09-14T06:01:16.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:16.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:16.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_31 2024-09-14T06:01:16.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:01:16.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:01:16.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_31 2024-09-14T06:01:16.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:16.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:16.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_31 2024-09-14T06:01:16.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_31 2024-09-14T06:01:16.262 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_31 2024-09-14T06:01:16.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:01:16.378+0000 7f1bc4aa5640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55710c78bd40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T06:01:16.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:01:16.378+0000 7f1bcbab3640 -1 rbd::mirror::ImageReplayer: 0x55710cdb8f00 [3/0b236112-fcc0-46d0-8e57-83f45f048a34] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T06:01:18.175 INFO:tasks.ceph.cluster2.osd.2.smithi191.stderr:2024-09-14T06:01:18.168+0000 7f3763311640 -1 reset not still connected to 0x55fad9e0f6c0 2024-09-14T06:01:18.176 INFO:tasks.ceph.cluster2.osd.2.smithi191.stderr:2024-09-14T06:01:18.171+0000 7f3763311640 -1 reset not still connected to 0x55fae63ef040 2024-09-14T06:01:18.176 INFO:tasks.ceph.cluster2.osd.2.smithi191.stderr:2024-09-14T06:01:18.172+0000 7f3763311640 -1 reset not still connected to 0x55fae63ef2b0 2024-09-14T06:01:18.176 INFO:tasks.ceph.cluster2.osd.2.smithi191.stderr:2024-09-14T06:01:18.172+0000 7f3763311640 -1 reset not still connected to 0x55faf0d5dc70 2024-09-14T06:01:18.178 INFO:tasks.ceph.cluster2.osd.2.smithi191.stderr:2024-09-14T06:01:18.178+0000 7f3763311640 -1 reset not still connected to 0x55faf4c43e10 2024-09-14T06:01:18.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:01:18.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:01:18.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:01:18.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_32 2024-09-14T06:01:18.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_32 snap 2024-09-14T06:01:18.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:18.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:18.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_32 2024-09-14T06:01:18.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-14T06:01:18.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_32@snap 2024-09-14T06:01:19.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T06:01:19.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_32 2024-09-14T06:01:19.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:19.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:19.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_32 2024-09-14T06:01:19.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:01:19.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:01:19.220 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_32 2024-09-14T06:01:19.220 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:19.221 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:19.221 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_32 2024-09-14T06:01:19.221 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_32 2024-09-14T06:01:19.273 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_32 2024-09-14T06:01:19.384 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:01:19.384+0000 7f1bc4aa5640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55710cca5ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T06:01:19.384 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:01:19.384+0000 7f1bcbab3640 -1 rbd::mirror::ImageReplayer: 0x55710d28e780 [3/bcdd7b18-22e7-4ae4-96b5-3c51260f96c7] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T06:01:21.326 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:01:21.329 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:01:21.330 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:01:21.330 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_33 2024-09-14T06:01:21.330 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_33 snap 2024-09-14T06:01:21.330 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:21.330 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:21.330 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_33 2024-09-14T06:01:21.330 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-14T06:01:21.330 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_33@snap 2024-09-14T06:01:22.205 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T06:01:22.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_33 2024-09-14T06:01:22.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:22.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:22.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_33 2024-09-14T06:01:22.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:01:22.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:01:22.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_33 2024-09-14T06:01:22.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:22.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:22.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_33 2024-09-14T06:01:22.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_33 2024-09-14T06:01:22.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_33 2024-09-14T06:01:22.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:01:22.395+0000 7f1bc52a6640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55710c78c000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T06:01:22.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:01:22.395+0000 7f1bcbab3640 -1 rbd::mirror::ImageReplayer: 0x55710d28e280 [3/abebcbd1-2502-4a74-8226-d2f7f8e8db34] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T06:01:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:01:24.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:01:24.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:01:24.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_34 2024-09-14T06:01:24.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_34 snap 2024-09-14T06:01:24.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:24.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:24.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_34 2024-09-14T06:01:24.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-14T06:01:24.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_34@snap 2024-09-14T06:01:25.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T06:01:25.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_34 2024-09-14T06:01:25.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:25.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:25.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_34 2024-09-14T06:01:25.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:01:25.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:01:25.225 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_34 2024-09-14T06:01:25.225 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:25.225 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:25.225 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_34 2024-09-14T06:01:25.225 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_34 2024-09-14T06:01:25.273 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_34 2024-09-14T06:01:25.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:01:25.391+0000 7f1bc52a6640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55710cca5860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T06:01:25.394 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:01:25.391+0000 7f1bcbab3640 -1 rbd::mirror::ImageReplayer: 0x55710d309680 [3/71b41da4-b167-4eb2-a0ea-67c283cfec10] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T06:01:27.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:01:27.341 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:01:27.341 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:01:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_35 2024-09-14T06:01:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_35 snap 2024-09-14T06:01:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_35 2024-09-14T06:01:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-14T06:01:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_35@snap 2024-09-14T06:01:28.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T06:01:28.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_35 2024-09-14T06:01:28.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:28.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:28.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_35 2024-09-14T06:01:28.165 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:01:28.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:01:28.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_35 2024-09-14T06:01:28.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:28.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:28.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_35 2024-09-14T06:01:28.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_35 2024-09-14T06:01:28.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_35 2024-09-14T06:01:28.340 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:01:28.339+0000 7f1bc4aa5640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55710ca7e4e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T06:01:28.340 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:01:28.339+0000 7f1bcbab3640 -1 rbd::mirror::ImageReplayer: 0x55710d308a00 [3/6249a2fd-1afb-4dee-89ca-39ba30cbe74d] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T06:01:30.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:01:30.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:01:30.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:01:30.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_36 2024-09-14T06:01:30.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_36 snap 2024-09-14T06:01:30.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:30.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:30.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_36 2024-09-14T06:01:30.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-14T06:01:30.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_36@snap 2024-09-14T06:01:31.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T06:01:31.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_36 2024-09-14T06:01:31.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:31.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:31.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_36 2024-09-14T06:01:31.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:01:31.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:01:31.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_36 2024-09-14T06:01:31.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:31.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:31.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_36 2024-09-14T06:01:31.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_36 2024-09-14T06:01:31.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_36 2024-09-14T06:01:31.641 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:01:31.640+0000 7f1bc52a6640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55710ca7e680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T06:01:31.641 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:01:31.640+0000 7f1bcbab3640 -1 rbd::mirror::ImageReplayer: 0x55710d309400 [3/702a468e-cfaa-4567-a91a-60fa43b31b1f] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T06:01:33.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:01:33.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:01:33.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:01:33.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_37 2024-09-14T06:01:33.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_37 snap 2024-09-14T06:01:33.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:33.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:33.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_37 2024-09-14T06:01:33.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-14T06:01:33.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_37@snap 2024-09-14T06:01:34.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T06:01:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_37 2024-09-14T06:01:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_37 2024-09-14T06:01:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:01:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:01:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_37 2024-09-14T06:01:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_37 2024-09-14T06:01:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_37 2024-09-14T06:01:34.313 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_37 2024-09-14T06:01:34.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:01:34.431+0000 7f1bc52a6640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55710c78b520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T06:01:34.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:01:34.431+0000 7f1bcbab3640 -1 rbd::mirror::ImageReplayer: 0x55710cdb8c80 [3/07ff7849-860f-4f87-8bed-88f824442785] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T06:01:36.375 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:01:36.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:01:36.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:01:36.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_38 2024-09-14T06:01:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_38 snap 2024-09-14T06:01:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_38 2024-09-14T06:01:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-14T06:01:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_38@snap 2024-09-14T06:01:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T06:01:37.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_38 2024-09-14T06:01:37.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:37.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:37.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_38 2024-09-14T06:01:37.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:01:37.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:01:37.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_38 2024-09-14T06:01:37.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:37.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:37.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_38 2024-09-14T06:01:37.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_38 2024-09-14T06:01:37.328 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_38 2024-09-14T06:01:37.442 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:01:37.441+0000 7f1bc52a6640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55710c877520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T06:01:37.442 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:01:37.441+0000 7f1bcb2b2640 -1 rbd::mirror::ImageReplayer: 0x55710d361680 [3/4cab209c-76f4-435e-9f27-921251cc9286] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T06:01:39.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:01:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:01:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:01:39.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_39 2024-09-14T06:01:39.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_39 snap 2024-09-14T06:01:39.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:39.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:39.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_39 2024-09-14T06:01:39.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-14T06:01:39.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_39@snap 2024-09-14T06:01:40.157 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T06:01:40.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_39 2024-09-14T06:01:40.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:40.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:40.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_39 2024-09-14T06:01:40.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:01:40.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:01:40.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_39 2024-09-14T06:01:40.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:40.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:40.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_39 2024-09-14T06:01:40.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_39 2024-09-14T06:01:40.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_39 2024-09-14T06:01:40.342 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:01:40.341+0000 7f1bc52a6640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55710c8776c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T06:01:40.342 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:01:40.341+0000 7f1bcb2b2640 -1 rbd::mirror::ImageReplayer: 0x55710d360a00 [3/37b3b339-d8a4-4dd0-ac91-56b0d1df8ebd] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T06:01:42.291 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:01:42.295 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:01:42.295 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:01:42.295 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_40 2024-09-14T06:01:42.296 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_40 snap 2024-09-14T06:01:42.296 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:42.296 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:42.296 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_40 2024-09-14T06:01:42.296 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-14T06:01:42.296 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_40@snap 2024-09-14T06:01:43.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T06:01:43.172 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_40 2024-09-14T06:01:43.172 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:43.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:43.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_40 2024-09-14T06:01:43.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:01:43.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:01:43.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_40 2024-09-14T06:01:43.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:43.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:43.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_40 2024-09-14T06:01:43.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_40 2024-09-14T06:01:43.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_40 2024-09-14T06:01:43.335 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:01:43.335+0000 7f1bc4aa5640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55710ca7e820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T06:01:43.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:01:43.335+0000 7f1bcbab3640 -1 rbd::mirror::ImageReplayer: 0x55710d360280 [3/5049b9ce-c032-4812-91d0-7dbca1b8b176] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T06:01:45.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:01:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:01:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:01:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_41 2024-09-14T06:01:45.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_41 snap 2024-09-14T06:01:45.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:45.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:45.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_41 2024-09-14T06:01:45.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-14T06:01:45.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_41@snap 2024-09-14T06:01:46.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T06:01:46.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_41 2024-09-14T06:01:46.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:46.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:46.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_41 2024-09-14T06:01:46.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:01:46.201 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:01:46.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_41 2024-09-14T06:01:46.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:46.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:46.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_41 2024-09-14T06:01:46.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_41 2024-09-14T06:01:46.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_41 2024-09-14T06:01:46.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:01:46.366+0000 7f1bc4aa5640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55710ca57520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T06:01:46.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:01:46.366+0000 7f1bcbab3640 -1 rbd::mirror::ImageReplayer: 0x55710d360000 [3/20523f16-1a82-463b-833f-79e70faaf271] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T06:01:48.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:01:48.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:01:48.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:01:48.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_42 2024-09-14T06:01:48.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_42 snap 2024-09-14T06:01:48.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_42 2024-09-14T06:01:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-14T06:01:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_42@snap 2024-09-14T06:01:49.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T06:01:49.225 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_42 2024-09-14T06:01:49.225 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:49.225 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:49.225 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_42 2024-09-14T06:01:49.225 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:01:49.226 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:01:49.226 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_42 2024-09-14T06:01:49.226 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:49.226 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:49.226 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_42 2024-09-14T06:01:49.226 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_42 2024-09-14T06:01:49.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_42 2024-09-14T06:01:49.394 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:01:49.393+0000 7f1bc4aa5640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55710c4f89c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T06:01:49.394 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:01:49.393+0000 7f1bcb2b2640 -1 rbd::mirror::ImageReplayer: 0x55710d308f00 [3/662fa768-1dea-4f15-91bf-edf297ebd84a] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T06:01:49.632 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.3354669259975108) > weight (0.25) 2024-09-14T06:01:49.633 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.2568628272748048) > weight (0.25) 2024-09-14T06:01:49.633 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.8020403124804462) > weight (0.25) 2024-09-14T06:01:49.633 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.9055744232497853) > weight (0.25) 2024-09-14T06:01:49.633 INFO:tasks.rbd_mirror_thrash:waiting for 95 secs before thrashing 2024-09-14T06:01:49.723 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-14T06:01:49.723 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-14T06:01:49.724 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-09-14T06:01:49.724 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-14T06:01:49.724+0000 7fcdbde8b640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 63035) UID: 1000 2024-09-14T06:01:51.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:01:51.324 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:01:51.324 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:01:51.324 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_43 2024-09-14T06:01:51.324 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_43 snap 2024-09-14T06:01:51.324 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:51.324 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:51.324 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_43 2024-09-14T06:01:51.324 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-14T06:01:51.324 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_43@snap 2024-09-14T06:01:52.200 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T06:01:52.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_43 2024-09-14T06:01:52.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:52.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:52.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_43 2024-09-14T06:01:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:01:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:01:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_43 2024-09-14T06:01:52.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_43 2024-09-14T06:01:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_43 2024-09-14T06:01:52.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_43 2024-09-14T06:01:52.394 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:01:52.393+0000 7f1bc52a6640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55710c4f81a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T06:01:52.394 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:01:52.393+0000 7f1bcbab3640 -1 rbd::mirror::ImageReplayer: 0x55710d28f180 [3/f71c9e7c-cbd0-4274-bae7-2c98cc5052b1] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T06:01:54.264 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:01:54.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:01:54.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:01:54.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_44 2024-09-14T06:01:54.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_44 snap 2024-09-14T06:01:54.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:54.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:54.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_44 2024-09-14T06:01:54.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-14T06:01:54.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_44@snap 2024-09-14T06:01:54.724 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-14T06:01:54.724 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T06:01:54.725 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-14T06:01:54.725 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-14T06:01:54.725 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-14T06:01:54.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-14T06:01:54.728 INFO:tasks.rbd_mirror_thrash:waiting for 75 secs before thrashing 2024-09-14T06:01:55.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T06:01:55.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_44 2024-09-14T06:01:55.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:55.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:55.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_44 2024-09-14T06:01:55.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:01:55.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:01:55.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_44 2024-09-14T06:01:55.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:55.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:55.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_44 2024-09-14T06:01:55.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_44 2024-09-14T06:01:55.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_44 2024-09-14T06:01:55.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:01:55.394+0000 7f1bc52a6640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55710ca571e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T06:01:55.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:01:55.394+0000 7f1bcb2b2640 -1 rbd::mirror::ImageReplayer: 0x55710d361180 [3/40153520-d46c-4ade-9ebf-442136d19014] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T06:01:57.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:01:57.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:01:57.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:01:57.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_45 2024-09-14T06:01:57.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_45 snap 2024-09-14T06:01:57.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:57.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:57.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_45 2024-09-14T06:01:57.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-14T06:01:57.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_45@snap 2024-09-14T06:01:58.170 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T06:01:58.190 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_45 2024-09-14T06:01:58.190 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:58.190 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:58.190 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_45 2024-09-14T06:01:58.190 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:01:58.190 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:01:58.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_45 2024-09-14T06:01:58.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:01:58.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:01:58.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_45 2024-09-14T06:01:58.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_45 2024-09-14T06:01:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_45 2024-09-14T06:01:58.352 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:01:58.352+0000 7f1bc52a6640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55710ca57380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T06:01:58.353 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:01:58.352+0000 7f1bcb2b2640 -1 rbd::mirror::ImageReplayer: 0x55710d33ef00 [3/8ed3596b-4b0c-421c-ae91-4a8f4c37bca1] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T06:02:00.341 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:02:00.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:00.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:02:00.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_46 2024-09-14T06:02:00.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_46 snap 2024-09-14T06:02:00.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:02:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_46 2024-09-14T06:02:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-14T06:02:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_46@snap 2024-09-14T06:02:01.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T06:02:01.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_46 2024-09-14T06:02:01.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:02:01.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:01.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_46 2024-09-14T06:02:01.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:02:01.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:02:01.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_46 2024-09-14T06:02:01.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:02:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_46 2024-09-14T06:02:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_46 2024-09-14T06:02:01.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_46 2024-09-14T06:02:01.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:02:01.413+0000 7f1bc52a6640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55710cd03380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T06:02:01.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:02:01.413+0000 7f1bcbab3640 -1 rbd::mirror::ImageReplayer: 0x55710d308000 [3/53467864-fd16-454f-8435-46f986b18a2f] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T06:02:03.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:02:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:02:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_47 2024-09-14T06:02:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_47 snap 2024-09-14T06:02:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:02:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_47 2024-09-14T06:02:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-14T06:02:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_47@snap 2024-09-14T06:02:04.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T06:02:04.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_47 2024-09-14T06:02:04.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:02:04.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:04.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_47 2024-09-14T06:02:04.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:02:04.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:02:04.262 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_47 2024-09-14T06:02:04.262 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:02:04.262 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:04.262 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_47 2024-09-14T06:02:04.262 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_47 2024-09-14T06:02:04.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_47 2024-09-14T06:02:04.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:02:04.436+0000 7f1bc4aa5640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55710cc81d40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T06:02:04.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:02:04.436+0000 7f1bcb2b2640 -1 rbd::mirror::ImageReplayer: 0x55710cdb9180 [3/0c24795a-37d5-4c8a-b3a8-c2ecda778c6a] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T06:02:06.363 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:02:06.366 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:06.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:02:06.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_48 2024-09-14T06:02:06.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_48 snap 2024-09-14T06:02:06.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:02:06.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:06.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_48 2024-09-14T06:02:06.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-14T06:02:06.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_48@snap 2024-09-14T06:02:07.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T06:02:07.270 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_48 2024-09-14T06:02:07.270 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:02:07.270 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:07.271 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_48 2024-09-14T06:02:07.271 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:02:07.271 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:02:07.272 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_48 2024-09-14T06:02:07.272 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:02:07.272 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:07.272 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_48 2024-09-14T06:02:07.272 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_48 2024-09-14T06:02:07.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_48 2024-09-14T06:02:07.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:02:07.434+0000 7f1bc52a6640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55710cd03860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T06:02:07.435 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:02:07.434+0000 7f1bcb2b2640 -1 rbd::mirror::ImageReplayer: 0x55710d308c80 [3/630369a5-7829-494e-9cd3-0d8a963a0ae4] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T06:02:09.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:02:09.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:09.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:02:09.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_49 2024-09-14T06:02:09.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_49 snap 2024-09-14T06:02:09.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:02:09.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:09.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_49 2024-09-14T06:02:09.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-14T06:02:09.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_49@snap 2024-09-14T06:02:10.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T06:02:10.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_49 2024-09-14T06:02:10.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:02:10.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:10.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_49 2024-09-14T06:02:10.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:02:10.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:02:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_49 2024-09-14T06:02:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:02:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_49 2024-09-14T06:02:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_49 2024-09-14T06:02:10.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_49 2024-09-14T06:02:10.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:02:10.456+0000 7f1bc4aa5640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55710cc84000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T06:02:10.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:02:10.456+0000 7f1bcb2b2640 -1 rbd::mirror::ImageReplayer: 0x55710cdb9b80 [3/2567c386-672b-4b12-ae9d-ec5af9fc0c43] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T06:02:12.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:02:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:02:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_50 2024-09-14T06:02:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_snapshot cluster2 mirror image_50 snap 2024-09-14T06:02:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:02:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_50 2024-09-14T06:02:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local snap=snap 2024-09-14T06:02:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap rm mirror/image_50@snap 2024-09-14T06:02:13.154 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing snap: 100% complete...done. 2024-09-14T06:02:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_50 2024-09-14T06:02:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:02:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_50 2024-09-14T06:02:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:02:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:02:13.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_50 2024-09-14T06:02:13.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:02:13.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:13.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_50 2024-09-14T06:02:13.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_50 2024-09-14T06:02:13.229 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_50 2024-09-14T06:02:13.350 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:02:13.349+0000 7f1bc52a6640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55710cd03520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T06:02:13.350 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:02:13.349+0000 7f1bcb2b2640 -1 rbd::mirror::ImageReplayer: 0x55710cdb8280 [3/c89c328d-a242-48de-b9a9-130c74b3f877] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T06:02:15.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-14T06:02:15.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:15.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ testlog 'TEST: image deletions should propagate' 2024-09-14T06:02:15.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ tee -a /tmp/tmp.sXArEbIrwv/rbd-mirror.test.log 2024-09-14T06:02:15.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ date '+%F %T' 2024-09-14T06:02:15.396 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo 2024-09-14 06:02:15 TEST: image deletions should propagate 2024-09-14T06:02:15.396 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:2024-09-14 06:02:15 TEST: image deletions should propagate 2024-09-14T06:02:15.396 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_pool_images cluster1 mirror 0 2024-09-14T06:02:15.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:15.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:15.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_count=0 2024-09-14T06:02:15.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:02:15.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local count 2024-09-14T06:02:15.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local last_count=0 2024-09-14T06:02:15.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ true 2024-09-14T06:02:15.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ seq 1 40 2024-09-14T06:02:15.398 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in `seq 1 40` 2024-09-14T06:02:15.398 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 1 -ne 1 2024-09-14T06:02:15.398 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-14T06:02:15.398 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep 'images: ' 2024-09-14T06:02:15.398 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cut '-d ' -f 2 2024-09-14T06:02:15.436 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: failed to query services: (13) Permission denied 2024-09-14T06:02:15.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ count=0 2024-09-14T06:02:15.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 0 = 0 2024-09-14T06:02:15.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:15.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_pool_healthy cluster1 mirror 0 2024-09-14T06:02:15.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:15.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:15.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:02:15.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state 2024-09-14T06:02:15.439 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ seq 1 40 2024-09-14T06:02:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in `seq 1 40` 2024-09-14T06:02:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 1 -ne 1 2024-09-14T06:02:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-14T06:02:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep 'image health:' 2024-09-14T06:02:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cut '-d ' -f 3 2024-09-14T06:02:15.477 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: failed to query services: (13) Permission denied 2024-09-14T06:02:15.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ state=OK 2024-09-14T06:02:15.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test OK = ERROR 2024-09-14T06:02:15.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test OK = OK 2024-09-14T06:02:15.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:15.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ seq 1 50 2024-09-14T06:02:15.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:02:15.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_1 2024-09-14T06:02:15.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2024-09-14T06:02:15.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:15.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:15.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-14T06:02:15.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:02:15.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:15.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:02:15.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_1 2024-09-14T06:02:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_1 2024-09-14T06:02:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-14T06:02:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-14T06:02:15.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:02:15.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:02:15.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:02:15.621 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2024-09-14T06:02:15.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:15.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:15.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-14T06:02:15.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:02:15.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:15.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:02:15.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:02:15.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_1 2024-09-14T06:02:15.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:15.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:15.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_1 2024-09-14T06:02:15.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-14T06:02:15.622 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:15.658 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-14T06:02:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:02:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:02:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:02:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_2 2024-09-14T06:02:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2024-09-14T06:02:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-14T06:02:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:02:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:02:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_2 2024-09-14T06:02:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_2 2024-09-14T06:02:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-14T06:02:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:15.697 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-14T06:02:15.700 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:02:15.700 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:02:15.700 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:02:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2024-09-14T06:02:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-14T06:02:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:02:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:02:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:02:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_2 2024-09-14T06:02:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_2 2024-09-14T06:02:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-14T06:02:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:15.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-14T06:02:15.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:02:15.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:15.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:02:15.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:15.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:02:15.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_3 2024-09-14T06:02:15.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2024-09-14T06:02:15.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:15.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:15.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-14T06:02:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:02:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:02:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_3 2024-09-14T06:02:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_3 2024-09-14T06:02:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-14T06:02:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:15.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-14T06:02:15.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:02:15.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:02:15.881 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:02:15.983 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2024-09-14T06:02:15.983 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:15.983 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:15.983 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-14T06:02:15.983 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:02:15.983 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:15.983 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:02:15.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:02:15.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_3 2024-09-14T06:02:15.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:15.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:15.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_3 2024-09-14T06:02:15.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-14T06:02:15.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:16.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-14T06:02:16.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:02:16.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:16.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:02:16.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:16.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:02:16.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_4 2024-09-14T06:02:16.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2024-09-14T06:02:16.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:16.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:16.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-14T06:02:16.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:02:16.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:16.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:02:16.028 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_4 2024-09-14T06:02:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_4 2024-09-14T06:02:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-14T06:02:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:16.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-14T06:02:16.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:02:16.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:02:16.066 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:02:16.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2024-09-14T06:02:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-14T06:02:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:02:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:02:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:02:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_4 2024-09-14T06:02:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_4 2024-09-14T06:02:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-14T06:02:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-14T06:02:16.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:02:16.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:16.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:02:16.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:16.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:02:16.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_5 2024-09-14T06:02:16.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2024-09-14T06:02:16.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-14T06:02:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:02:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:02:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_5 2024-09-14T06:02:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:16.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_5 2024-09-14T06:02:16.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-14T06:02:16.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-14T06:02:16.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:02:16.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:02:16.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:02:16.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2024-09-14T06:02:16.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:16.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:16.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-14T06:02:16.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:02:16.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:16.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:02:16.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:02:16.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_5 2024-09-14T06:02:16.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:16.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:16.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_5 2024-09-14T06:02:16.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-14T06:02:16.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:16.383 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-14T06:02:16.387 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:02:16.387 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:16.387 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:02:16.387 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:16.387 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:02:16.387 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_6 2024-09-14T06:02:16.387 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2024-09-14T06:02:16.387 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:16.387 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:16.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-14T06:02:16.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:02:16.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:16.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:02:16.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:16.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_6 2024-09-14T06:02:16.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:16.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:16.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_6 2024-09-14T06:02:16.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-14T06:02:16.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:16.422 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-09-14T06:02:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:02:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:02:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:02:16.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2024-09-14T06:02:16.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:16.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:16.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-14T06:02:16.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:02:16.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:16.528 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:02:16.528 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:02:16.528 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_6 2024-09-14T06:02:16.528 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:16.528 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:16.528 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_6 2024-09-14T06:02:16.528 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-14T06:02:16.528 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:16.562 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-09-14T06:02:16.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:02:16.566 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:02:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:02:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_7 2024-09-14T06:02:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2024-09-14T06:02:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-14T06:02:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:02:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:02:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_7 2024-09-14T06:02:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_7 2024-09-14T06:02:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-14T06:02:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:16.601 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-14T06:02:16.604 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:02:16.604 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:02:16.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:02:16.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2024-09-14T06:02:16.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:16.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:16.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-14T06:02:16.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:02:16.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:16.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:02:16.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:02:16.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_7 2024-09-14T06:02:16.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:16.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:16.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_7 2024-09-14T06:02:16.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-14T06:02:16.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:16.742 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-14T06:02:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:02:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:02:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:02:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_8 2024-09-14T06:02:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2024-09-14T06:02:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-14T06:02:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:02:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:02:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_8 2024-09-14T06:02:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_8 2024-09-14T06:02:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-14T06:02:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:16.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-14T06:02:16.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:02:16.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:02:16.988 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:02:17.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2024-09-14T06:02:17.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:17.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:17.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-14T06:02:17.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:02:17.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:17.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:02:17.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:02:17.091 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_8 2024-09-14T06:02:17.091 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:17.091 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:17.091 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_8 2024-09-14T06:02:17.091 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-14T06:02:17.091 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:17.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-14T06:02:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:02:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:02:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:17.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:02:17.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_9 2024-09-14T06:02:17.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2024-09-14T06:02:17.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:17.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:17.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-14T06:02:17.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:02:17.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:17.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:02:17.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:17.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_9 2024-09-14T06:02:17.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:17.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:17.132 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_9 2024-09-14T06:02:17.132 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-14T06:02:17.132 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:17.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-14T06:02:17.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:02:17.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:02:17.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:02:17.271 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2024-09-14T06:02:17.271 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:17.271 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:17.271 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-14T06:02:17.271 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:02:17.271 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:17.271 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:02:17.271 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:02:17.271 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_9 2024-09-14T06:02:17.271 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:17.271 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:17.272 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_9 2024-09-14T06:02:17.272 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-14T06:02:17.272 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:17.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-14T06:02:17.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:02:17.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:17.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:02:17.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:17.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:02:17.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_10 2024-09-14T06:02:17.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2024-09-14T06:02:17.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:17.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:17.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-14T06:02:17.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:02:17.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:17.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:02:17.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:17.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_10 2024-09-14T06:02:17.312 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:17.312 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:17.312 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_10 2024-09-14T06:02:17.312 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-14T06:02:17.312 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:17.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-14T06:02:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:02:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:02:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:02:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2024-09-14T06:02:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-14T06:02:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:02:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:02:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:02:17.452 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_10 2024-09-14T06:02:17.452 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:17.452 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:17.452 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_10 2024-09-14T06:02:17.452 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-14T06:02:17.452 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-14T06:02:17.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:02:17.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:17.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:02:17.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:17.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:02:17.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_11 2024-09-14T06:02:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_11 deleted 2024-09-14T06:02:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_11 2024-09-14T06:02:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:02:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:02:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_11 2024-09-14T06:02:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_11 2024-09-14T06:02:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2024-09-14T06:02:17.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:17.526 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_11: (2) No such file or directory 2024-09-14T06:02:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:02:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:02:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:02:17.631 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_11 deleted '' 2024-09-14T06:02:17.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:17.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:17.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_11 2024-09-14T06:02:17.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:02:17.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:17.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:02:17.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:02:17.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_11 2024-09-14T06:02:17.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:17.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:17.632 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_11 2024-09-14T06:02:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2024-09-14T06:02:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:17.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_11: (2) No such file or directory 2024-09-14T06:02:17.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:02:17.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:17.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:02:17.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:17.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:02:17.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_12 2024-09-14T06:02:17.671 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_12 deleted 2024-09-14T06:02:17.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:17.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:17.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_12 2024-09-14T06:02:17.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:02:17.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:17.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:02:17.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:17.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_12 2024-09-14T06:02:17.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:17.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:17.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_12 2024-09-14T06:02:17.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2024-09-14T06:02:17.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_12: (2) No such file or directory 2024-09-14T06:02:17.709 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:02:17.710 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:02:17.710 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:02:17.811 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_12 deleted '' 2024-09-14T06:02:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_12 2024-09-14T06:02:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:02:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:02:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:02:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_12 2024-09-14T06:02:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_12 2024-09-14T06:02:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2024-09-14T06:02:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_12: (2) No such file or directory 2024-09-14T06:02:17.852 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:02:17.852 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:17.852 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:02:17.852 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:17.852 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:02:17.852 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_13 2024-09-14T06:02:17.852 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_13 deleted 2024-09-14T06:02:17.852 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:17.852 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:17.852 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_13 2024-09-14T06:02:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:02:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:02:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_13 2024-09-14T06:02:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_13 2024-09-14T06:02:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2024-09-14T06:02:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:17.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_13: (2) No such file or directory 2024-09-14T06:02:17.891 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:02:17.891 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:02:17.891 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:02:17.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_13 deleted '' 2024-09-14T06:02:17.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:17.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:17.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_13 2024-09-14T06:02:17.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:02:17.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:17.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:02:17.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:02:17.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_13 2024-09-14T06:02:17.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:17.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:17.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_13 2024-09-14T06:02:17.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2024-09-14T06:02:17.994 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:18.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_13: (2) No such file or directory 2024-09-14T06:02:18.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:02:18.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:18.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:02:18.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:18.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:02:18.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_14 2024-09-14T06:02:18.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_14 deleted 2024-09-14T06:02:18.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:18.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:18.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_14 2024-09-14T06:02:18.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:02:18.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:18.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:02:18.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:18.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_14 2024-09-14T06:02:18.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:18.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:18.034 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_14 2024-09-14T06:02:18.035 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2024-09-14T06:02:18.035 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:18.067 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_14: (2) No such file or directory 2024-09-14T06:02:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:02:18.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:02:18.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:02:18.172 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_14 deleted '' 2024-09-14T06:02:18.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:18.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:18.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_14 2024-09-14T06:02:18.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:02:18.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:18.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:02:18.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:02:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_14 2024-09-14T06:02:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_14 2024-09-14T06:02:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2024-09-14T06:02:18.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_14: (2) No such file or directory 2024-09-14T06:02:18.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:02:18.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:18.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:02:18.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:18.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:02:18.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_15 2024-09-14T06:02:18.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_15 deleted 2024-09-14T06:02:18.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:18.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:18.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_15 2024-09-14T06:02:18.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:02:18.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:18.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:02:18.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:18.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_15 2024-09-14T06:02:18.214 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:18.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:18.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_15 2024-09-14T06:02:18.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2024-09-14T06:02:18.215 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:18.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_15: (2) No such file or directory 2024-09-14T06:02:18.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:02:18.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:02:18.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:02:18.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_15 deleted '' 2024-09-14T06:02:18.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:18.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:18.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_15 2024-09-14T06:02:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:02:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:02:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:02:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_15 2024-09-14T06:02:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_15 2024-09-14T06:02:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2024-09-14T06:02:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:18.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_15: (2) No such file or directory 2024-09-14T06:02:18.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:02:18.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:18.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:02:18.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:18.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:02:18.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_16 2024-09-14T06:02:18.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_16 deleted 2024-09-14T06:02:18.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:18.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:18.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_16 2024-09-14T06:02:18.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:02:18.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:18.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:02:18.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:18.396 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_16 2024-09-14T06:02:18.396 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:18.396 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:18.396 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_16 2024-09-14T06:02:18.396 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2024-09-14T06:02:18.396 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:18.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_16: (2) No such file or directory 2024-09-14T06:02:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:02:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:02:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:02:18.534 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_16 deleted '' 2024-09-14T06:02:18.534 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:18.534 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:18.534 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_16 2024-09-14T06:02:18.534 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:02:18.534 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:18.534 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:02:18.534 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:02:18.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_16 2024-09-14T06:02:18.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:18.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:18.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_16 2024-09-14T06:02:18.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2024-09-14T06:02:18.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:18.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_16: (2) No such file or directory 2024-09-14T06:02:18.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:02:18.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:18.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:02:18.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:18.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:02:18.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_17 2024-09-14T06:02:18.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_17 deleted 2024-09-14T06:02:18.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:18.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:18.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_17 2024-09-14T06:02:18.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:02:18.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:18.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:02:18.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:18.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_17 2024-09-14T06:02:18.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:18.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:18.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_17 2024-09-14T06:02:18.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2024-09-14T06:02:18.575 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:18.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_17: (2) No such file or directory 2024-09-14T06:02:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:02:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:02:18.612 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:02:18.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_17 deleted '' 2024-09-14T06:02:18.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:18.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:18.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_17 2024-09-14T06:02:18.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:02:18.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:18.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:02:18.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:02:18.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_17 2024-09-14T06:02:18.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:18.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:18.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_17 2024-09-14T06:02:18.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2024-09-14T06:02:18.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:18.748 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_17: (2) No such file or directory 2024-09-14T06:02:18.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:02:18.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:18.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:02:18.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:18.752 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:02:18.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_18 2024-09-14T06:02:18.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_18 deleted 2024-09-14T06:02:18.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:18.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:18.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_18 2024-09-14T06:02:18.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:02:18.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:18.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:02:18.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:18.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_18 2024-09-14T06:02:18.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:18.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:18.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_18 2024-09-14T06:02:18.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2024-09-14T06:02:18.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:18.787 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_18: (2) No such file or directory 2024-09-14T06:02:18.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:02:18.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:02:18.791 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:02:18.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_18 deleted '' 2024-09-14T06:02:18.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:18.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:18.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_18 2024-09-14T06:02:18.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:02:18.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:18.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:02:18.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:02:18.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_18 2024-09-14T06:02:18.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:18.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:18.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_18 2024-09-14T06:02:18.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2024-09-14T06:02:18.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:18.930 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_18: (2) No such file or directory 2024-09-14T06:02:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:02:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:02:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:02:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_19 2024-09-14T06:02:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_19 deleted 2024-09-14T06:02:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_19 2024-09-14T06:02:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:02:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:02:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_19 2024-09-14T06:02:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:18.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:18.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_19 2024-09-14T06:02:18.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2024-09-14T06:02:18.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:18.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_19: (2) No such file or directory 2024-09-14T06:02:18.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:02:18.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:02:18.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:02:19.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_19 deleted '' 2024-09-14T06:02:19.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:19.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:19.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_19 2024-09-14T06:02:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:02:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:02:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:02:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_19 2024-09-14T06:02:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_19 2024-09-14T06:02:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2024-09-14T06:02:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:19.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_19: (2) No such file or directory 2024-09-14T06:02:19.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:02:19.124 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:19.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:02:19.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:19.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:02:19.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_20 2024-09-14T06:02:19.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_20 deleted 2024-09-14T06:02:19.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:19.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:19.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_20 2024-09-14T06:02:19.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:02:19.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:19.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:02:19.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:19.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_20 2024-09-14T06:02:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_20 2024-09-14T06:02:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2024-09-14T06:02:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:19.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_20: (2) No such file or directory 2024-09-14T06:02:19.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:02:19.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:02:19.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:02:19.264 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_20 deleted '' 2024-09-14T06:02:19.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:19.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:19.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_20 2024-09-14T06:02:19.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:02:19.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:19.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:02:19.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:02:19.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_20 2024-09-14T06:02:19.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:19.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:19.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_20 2024-09-14T06:02:19.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2024-09-14T06:02:19.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:19.299 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_20: (2) No such file or directory 2024-09-14T06:02:19.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:02:19.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:19.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:02:19.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:19.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:02:19.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_21 2024-09-14T06:02:19.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_21 deleted 2024-09-14T06:02:19.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:19.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:19.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_21 2024-09-14T06:02:19.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:02:19.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:19.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:02:19.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:19.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_21 2024-09-14T06:02:19.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:19.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:19.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_21 2024-09-14T06:02:19.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2024-09-14T06:02:19.305 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:19.338 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_21: (2) No such file or directory 2024-09-14T06:02:19.341 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:02:19.341 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:02:19.341 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:02:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_21 deleted '' 2024-09-14T06:02:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_21 2024-09-14T06:02:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:02:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:02:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:02:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_21 2024-09-14T06:02:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_21 2024-09-14T06:02:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2024-09-14T06:02:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:19.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_21: (2) No such file or directory 2024-09-14T06:02:19.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:02:19.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:19.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:02:19.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:19.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:02:19.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_22 2024-09-14T06:02:19.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_22 deleted 2024-09-14T06:02:19.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:19.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:19.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_22 2024-09-14T06:02:19.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:02:19.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:19.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:02:19.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:19.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_22 2024-09-14T06:02:19.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:19.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:19.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_22 2024-09-14T06:02:19.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2024-09-14T06:02:19.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_22: (2) No such file or directory 2024-09-14T06:02:19.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:02:19.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:02:19.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:02:19.623 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_22 deleted '' 2024-09-14T06:02:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_22 2024-09-14T06:02:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:02:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:02:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:02:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_22 2024-09-14T06:02:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_22 2024-09-14T06:02:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2024-09-14T06:02:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:19.659 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_22: (2) No such file or directory 2024-09-14T06:02:19.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:02:19.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:19.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:02:19.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:19.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:02:19.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_23 2024-09-14T06:02:19.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_23 deleted 2024-09-14T06:02:19.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:19.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:19.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_23 2024-09-14T06:02:19.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:02:19.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:19.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:02:19.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:19.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_23 2024-09-14T06:02:19.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:19.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:19.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_23 2024-09-14T06:02:19.664 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2024-09-14T06:02:19.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:19.699 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_23: (2) No such file or directory 2024-09-14T06:02:19.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:02:19.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:02:19.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:02:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_23 deleted '' 2024-09-14T06:02:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_23 2024-09-14T06:02:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:02:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:02:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:02:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_23 2024-09-14T06:02:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_23 2024-09-14T06:02:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2024-09-14T06:02:19.805 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_23: (2) No such file or directory 2024-09-14T06:02:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:02:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:02:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:02:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_24 2024-09-14T06:02:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_24 deleted 2024-09-14T06:02:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_24 2024-09-14T06:02:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:02:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:02:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:19.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_24 2024-09-14T06:02:19.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:19.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:19.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_24 2024-09-14T06:02:19.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2024-09-14T06:02:19.846 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:19.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_24: (2) No such file or directory 2024-09-14T06:02:19.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:02:19.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:02:19.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:02:19.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_24 deleted '' 2024-09-14T06:02:19.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:19.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:19.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_24 2024-09-14T06:02:19.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:02:19.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:19.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:02:19.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:02:19.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_24 2024-09-14T06:02:19.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:19.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:19.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_24 2024-09-14T06:02:19.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2024-09-14T06:02:19.985 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:20.021 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_24: (2) No such file or directory 2024-09-14T06:02:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:02:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:02:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:02:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_25 2024-09-14T06:02:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_25 deleted 2024-09-14T06:02:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_25 2024-09-14T06:02:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:02:20.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:20.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:02:20.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:20.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_25 2024-09-14T06:02:20.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:20.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:20.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_25 2024-09-14T06:02:20.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2024-09-14T06:02:20.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:20.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_25: (2) No such file or directory 2024-09-14T06:02:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:02:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:02:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:02:20.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_25 deleted '' 2024-09-14T06:02:20.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:20.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:20.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_25 2024-09-14T06:02:20.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:02:20.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:20.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:02:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:02:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_25 2024-09-14T06:02:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_25 2024-09-14T06:02:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2024-09-14T06:02:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:20.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_25: (2) No such file or directory 2024-09-14T06:02:20.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:02:20.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:20.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:02:20.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:20.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:02:20.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_26 2024-09-14T06:02:20.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_26 deleted 2024-09-14T06:02:20.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:20.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:20.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_26 2024-09-14T06:02:20.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:02:20.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:20.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:02:20.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:20.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_26 2024-09-14T06:02:20.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:20.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:20.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_26 2024-09-14T06:02:20.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2024-09-14T06:02:20.208 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:20.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_26: (2) No such file or directory 2024-09-14T06:02:20.245 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:02:20.246 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:02:20.246 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:02:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_26 deleted '' 2024-09-14T06:02:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_26 2024-09-14T06:02:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:02:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:02:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:02:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_26 2024-09-14T06:02:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:20.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:20.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_26 2024-09-14T06:02:20.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2024-09-14T06:02:20.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:20.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_26: (2) No such file or directory 2024-09-14T06:02:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:02:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:02:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:02:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_27 2024-09-14T06:02:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_27 deleted 2024-09-14T06:02:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_27 2024-09-14T06:02:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:02:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:02:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_27 2024-09-14T06:02:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_27 2024-09-14T06:02:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2024-09-14T06:02:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:20.424 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_27: (2) No such file or directory 2024-09-14T06:02:20.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:02:20.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:02:20.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:02:20.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_27 deleted '' 2024-09-14T06:02:20.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:20.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:20.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_27 2024-09-14T06:02:20.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:02:20.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:20.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:02:20.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:02:20.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_27 2024-09-14T06:02:20.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:20.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:20.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_27 2024-09-14T06:02:20.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2024-09-14T06:02:20.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:20.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_27: (2) No such file or directory 2024-09-14T06:02:20.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:02:20.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:20.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:02:20.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:20.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:02:20.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_28 2024-09-14T06:02:20.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_28 deleted 2024-09-14T06:02:20.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:20.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_28 2024-09-14T06:02:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:02:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:02:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_28 2024-09-14T06:02:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_28 2024-09-14T06:02:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2024-09-14T06:02:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:20.608 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_28: (2) No such file or directory 2024-09-14T06:02:20.611 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:02:20.611 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:02:20.611 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:02:20.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_28 deleted '' 2024-09-14T06:02:20.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:20.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:20.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_28 2024-09-14T06:02:20.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:02:20.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:02:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:02:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_28 2024-09-14T06:02:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_28 2024-09-14T06:02:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2024-09-14T06:02:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:20.749 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_28: (2) No such file or directory 2024-09-14T06:02:20.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:02:20.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:20.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:02:20.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:20.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:02:20.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_29 2024-09-14T06:02:20.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_29 deleted 2024-09-14T06:02:20.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:20.754 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:20.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_29 2024-09-14T06:02:20.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:02:20.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:20.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:02:20.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:20.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_29 2024-09-14T06:02:20.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:20.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:20.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_29 2024-09-14T06:02:20.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2024-09-14T06:02:20.755 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:20.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_29: (2) No such file or directory 2024-09-14T06:02:20.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:02:20.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:02:20.793 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:02:20.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_29 deleted '' 2024-09-14T06:02:20.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:20.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:20.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_29 2024-09-14T06:02:20.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:02:20.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:20.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:02:20.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:02:20.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_29 2024-09-14T06:02:20.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:20.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:20.894 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_29 2024-09-14T06:02:20.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2024-09-14T06:02:20.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:20.930 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_29: (2) No such file or directory 2024-09-14T06:02:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:02:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:02:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:02:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_30 2024-09-14T06:02:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_30 deleted 2024-09-14T06:02:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:20.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:20.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_30 2024-09-14T06:02:20.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:02:20.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:20.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:02:20.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:20.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_30 2024-09-14T06:02:20.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:20.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:20.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_30 2024-09-14T06:02:20.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2024-09-14T06:02:20.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:20.969 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_30: (2) No such file or directory 2024-09-14T06:02:20.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:02:20.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:02:20.973 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:02:21.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_30 deleted '' 2024-09-14T06:02:21.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:21.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:21.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_30 2024-09-14T06:02:21.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:02:21.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:21.075 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:02:21.076 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:02:21.076 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_30 2024-09-14T06:02:21.076 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:21.076 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:21.076 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_30 2024-09-14T06:02:21.076 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2024-09-14T06:02:21.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:21.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_30: (2) No such file or directory 2024-09-14T06:02:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:02:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:02:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:02:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_31 2024-09-14T06:02:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_31 deleted 2024-09-14T06:02:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_31 2024-09-14T06:02:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:02:21.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:21.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:02:21.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:21.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_31 2024-09-14T06:02:21.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:21.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:21.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_31 2024-09-14T06:02:21.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2024-09-14T06:02:21.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:21.150 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_31: (2) No such file or directory 2024-09-14T06:02:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:02:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:02:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:02:21.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_31 deleted '' 2024-09-14T06:02:21.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:21.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:21.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_31 2024-09-14T06:02:21.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:02:21.255 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:21.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:02:21.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:02:21.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_31 2024-09-14T06:02:21.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:21.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:21.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_31 2024-09-14T06:02:21.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2024-09-14T06:02:21.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:21.292 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_31: (2) No such file or directory 2024-09-14T06:02:21.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:02:21.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:21.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:02:21.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:21.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:02:21.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_32 2024-09-14T06:02:21.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_32 deleted 2024-09-14T06:02:21.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:21.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:21.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_32 2024-09-14T06:02:21.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:02:21.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:21.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:02:21.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:21.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_32 2024-09-14T06:02:21.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:21.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:21.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_32 2024-09-14T06:02:21.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2024-09-14T06:02:21.298 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:21.331 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_32: (2) No such file or directory 2024-09-14T06:02:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:02:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:02:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:02:21.437 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_32 deleted '' 2024-09-14T06:02:21.437 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:21.437 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:21.437 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_32 2024-09-14T06:02:21.437 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:02:21.437 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:21.437 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:02:21.437 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:02:21.437 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_32 2024-09-14T06:02:21.437 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:21.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:21.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_32 2024-09-14T06:02:21.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2024-09-14T06:02:21.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:21.473 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_32: (2) No such file or directory 2024-09-14T06:02:21.478 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:02:21.478 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:21.478 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:02:21.478 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:21.478 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:02:21.478 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_33 2024-09-14T06:02:21.478 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_33 deleted 2024-09-14T06:02:21.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:21.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:21.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_33 2024-09-14T06:02:21.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:02:21.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:21.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:02:21.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:21.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_33 2024-09-14T06:02:21.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:21.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:21.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_33 2024-09-14T06:02:21.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2024-09-14T06:02:21.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_33: (2) No such file or directory 2024-09-14T06:02:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:02:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:02:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:02:21.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_33 deleted '' 2024-09-14T06:02:21.821 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:21.821 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:21.821 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_33 2024-09-14T06:02:21.821 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:02:21.821 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:21.821 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:02:21.822 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:02:21.822 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_33 2024-09-14T06:02:21.822 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:21.822 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:21.822 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_33 2024-09-14T06:02:21.822 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2024-09-14T06:02:21.822 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:21.856 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_33: (2) No such file or directory 2024-09-14T06:02:21.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:02:21.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:21.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:02:21.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:21.860 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:02:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_34 2024-09-14T06:02:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_34 deleted 2024-09-14T06:02:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_34 2024-09-14T06:02:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:02:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:02:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_34 2024-09-14T06:02:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_34 2024-09-14T06:02:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2024-09-14T06:02:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:21.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_34: (2) No such file or directory 2024-09-14T06:02:21.899 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:02:21.899 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:02:21.899 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:02:22.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_34 deleted '' 2024-09-14T06:02:22.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:22.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:22.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_34 2024-09-14T06:02:22.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:02:22.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:22.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:02:22.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:02:22.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_34 2024-09-14T06:02:22.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:22.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:22.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_34 2024-09-14T06:02:22.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2024-09-14T06:02:22.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:22.037 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_34: (2) No such file or directory 2024-09-14T06:02:22.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:02:22.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:22.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:02:22.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:22.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:02:22.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_35 2024-09-14T06:02:22.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_35 deleted 2024-09-14T06:02:22.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:22.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:22.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_35 2024-09-14T06:02:22.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:02:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:02:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_35 2024-09-14T06:02:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_35 2024-09-14T06:02:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2024-09-14T06:02:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:22.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_35: (2) No such file or directory 2024-09-14T06:02:22.080 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:02:22.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:02:22.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:02:22.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_35 deleted '' 2024-09-14T06:02:22.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:22.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:22.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_35 2024-09-14T06:02:22.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:02:22.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:22.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:02:22.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:02:22.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_35 2024-09-14T06:02:22.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:22.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:22.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_35 2024-09-14T06:02:22.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2024-09-14T06:02:22.184 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:22.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_35: (2) No such file or directory 2024-09-14T06:02:22.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:02:22.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:22.225 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:02:22.225 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:22.225 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:02:22.225 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_36 2024-09-14T06:02:22.225 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_36 deleted 2024-09-14T06:02:22.225 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:22.225 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:22.225 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_36 2024-09-14T06:02:22.225 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:02:22.225 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:22.225 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:02:22.225 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:22.226 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_36 2024-09-14T06:02:22.226 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:22.226 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:22.226 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_36 2024-09-14T06:02:22.226 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2024-09-14T06:02:22.226 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:22.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_36: (2) No such file or directory 2024-09-14T06:02:22.262 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:02:22.262 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:02:22.262 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:02:22.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_36 deleted '' 2024-09-14T06:02:22.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:22.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:22.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_36 2024-09-14T06:02:22.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:02:22.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:22.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:02:22.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:02:22.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_36 2024-09-14T06:02:22.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:22.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:22.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_36 2024-09-14T06:02:22.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2024-09-14T06:02:22.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:22.401 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_36: (2) No such file or directory 2024-09-14T06:02:22.406 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:02:22.406 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:22.407 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:02:22.407 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:22.407 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:02:22.407 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_37 2024-09-14T06:02:22.407 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_37 deleted 2024-09-14T06:02:22.407 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:22.407 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:22.408 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_37 2024-09-14T06:02:22.408 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:02:22.408 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:22.408 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:02:22.408 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:22.408 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_37 2024-09-14T06:02:22.408 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:22.408 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:22.408 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_37 2024-09-14T06:02:22.408 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2024-09-14T06:02:22.408 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:22.440 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_37: (2) No such file or directory 2024-09-14T06:02:22.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:02:22.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:02:22.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:02:22.545 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_37 deleted '' 2024-09-14T06:02:22.545 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:22.545 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:22.545 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_37 2024-09-14T06:02:22.546 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:02:22.546 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:22.546 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:02:22.546 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:02:22.546 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_37 2024-09-14T06:02:22.546 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:22.546 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:22.546 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_37 2024-09-14T06:02:22.546 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2024-09-14T06:02:22.546 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:22.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_37: (2) No such file or directory 2024-09-14T06:02:22.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:02:22.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:22.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:02:22.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:22.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:02:22.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_38 2024-09-14T06:02:22.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_38 deleted 2024-09-14T06:02:22.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:22.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:22.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_38 2024-09-14T06:02:22.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:02:22.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:22.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:02:22.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:22.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_38 2024-09-14T06:02:22.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:22.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:22.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_38 2024-09-14T06:02:22.588 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2024-09-14T06:02:22.588 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:22.621 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_38: (2) No such file or directory 2024-09-14T06:02:22.624 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:02:22.624 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:02:22.624 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:02:22.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_38 deleted '' 2024-09-14T06:02:22.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:22.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:22.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_38 2024-09-14T06:02:22.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:02:22.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:22.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:02:22.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:02:22.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_38 2024-09-14T06:02:22.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:22.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:22.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_38 2024-09-14T06:02:22.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2024-09-14T06:02:22.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_38: (2) No such file or directory 2024-09-14T06:02:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:02:22.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:22.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:02:22.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:22.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:02:22.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_39 2024-09-14T06:02:22.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_39 deleted 2024-09-14T06:02:22.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:22.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:22.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_39 2024-09-14T06:02:22.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:02:22.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:22.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:02:22.766 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:22.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_39 2024-09-14T06:02:22.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:22.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:22.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_39 2024-09-14T06:02:22.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2024-09-14T06:02:22.767 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:22.801 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_39: (2) No such file or directory 2024-09-14T06:02:22.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:02:22.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:02:22.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:02:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_39 deleted '' 2024-09-14T06:02:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_39 2024-09-14T06:02:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:02:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:02:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:02:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_39 2024-09-14T06:02:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_39 2024-09-14T06:02:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2024-09-14T06:02:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:22.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_39: (2) No such file or directory 2024-09-14T06:02:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:02:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:02:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:02:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_40 2024-09-14T06:02:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_40 deleted 2024-09-14T06:02:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_40 2024-09-14T06:02:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:02:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:22.947 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:02:22.947 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:22.947 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_40 2024-09-14T06:02:22.947 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:22.947 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:22.947 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_40 2024-09-14T06:02:22.947 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2024-09-14T06:02:22.947 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_40: (2) No such file or directory 2024-09-14T06:02:22.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:02:22.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:02:22.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:02:23.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_40 deleted '' 2024-09-14T06:02:23.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:23.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:23.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_40 2024-09-14T06:02:23.086 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:02:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:02:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:02:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_40 2024-09-14T06:02:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_40 2024-09-14T06:02:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2024-09-14T06:02:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:23.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_40: (2) No such file or directory 2024-09-14T06:02:23.126 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:02:23.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:23.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:02:23.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:23.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:02:23.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_41 2024-09-14T06:02:23.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_41 deleted 2024-09-14T06:02:23.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:23.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:23.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_41 2024-09-14T06:02:23.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:02:23.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:23.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:02:23.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_41 2024-09-14T06:02:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_41 2024-09-14T06:02:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2024-09-14T06:02:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:23.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_41: (2) No such file or directory 2024-09-14T06:02:23.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:02:23.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:02:23.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:02:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_41 deleted '' 2024-09-14T06:02:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:23.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_41 2024-09-14T06:02:23.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:02:23.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:23.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:02:23.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:02:23.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_41 2024-09-14T06:02:23.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:23.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:23.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_41 2024-09-14T06:02:23.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2024-09-14T06:02:23.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:23.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_41: (2) No such file or directory 2024-09-14T06:02:23.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:02:23.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:23.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:02:23.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:23.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:02:23.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_42 2024-09-14T06:02:23.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_42 deleted 2024-09-14T06:02:23.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:23.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:23.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_42 2024-09-14T06:02:23.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:02:23.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:23.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:02:23.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:23.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_42 2024-09-14T06:02:23.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:23.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:23.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_42 2024-09-14T06:02:23.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2024-09-14T06:02:23.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:23.343 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_42: (2) No such file or directory 2024-09-14T06:02:23.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:02:23.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:02:23.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:02:23.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_42 deleted '' 2024-09-14T06:02:23.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:23.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:23.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_42 2024-09-14T06:02:23.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:02:23.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:23.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:02:23.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:02:23.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_42 2024-09-14T06:02:23.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:23.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:23.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_42 2024-09-14T06:02:23.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2024-09-14T06:02:23.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:23.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_42: (2) No such file or directory 2024-09-14T06:02:23.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:02:23.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:23.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:02:23.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:23.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:02:23.491 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_43 2024-09-14T06:02:23.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_43 deleted 2024-09-14T06:02:23.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:23.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:23.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_43 2024-09-14T06:02:23.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:02:23.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:23.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:02:23.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:23.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_43 2024-09-14T06:02:23.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:23.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:23.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_43 2024-09-14T06:02:23.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2024-09-14T06:02:23.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:23.524 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_43: (2) No such file or directory 2024-09-14T06:02:23.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:02:23.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:02:23.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:02:23.629 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_43 deleted '' 2024-09-14T06:02:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_43 2024-09-14T06:02:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:02:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:02:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:02:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_43 2024-09-14T06:02:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_43 2024-09-14T06:02:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2024-09-14T06:02:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:23.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_43: (2) No such file or directory 2024-09-14T06:02:23.669 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:02:23.669 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:23.669 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:02:23.669 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:23.669 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:02:23.669 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_44 2024-09-14T06:02:23.669 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_44 deleted 2024-09-14T06:02:23.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:23.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:23.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_44 2024-09-14T06:02:23.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:02:23.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:23.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:02:23.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:23.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_44 2024-09-14T06:02:23.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:23.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:23.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_44 2024-09-14T06:02:23.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2024-09-14T06:02:23.670 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:23.704 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_44: (2) No such file or directory 2024-09-14T06:02:23.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:02:23.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:02:23.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:02:23.808 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_44 deleted '' 2024-09-14T06:02:23.808 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_44 2024-09-14T06:02:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:02:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:02:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:02:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_44 2024-09-14T06:02:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_44 2024-09-14T06:02:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2024-09-14T06:02:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:23.844 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_44: (2) No such file or directory 2024-09-14T06:02:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:02:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:02:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:23.848 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:02:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_45 2024-09-14T06:02:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_45 deleted 2024-09-14T06:02:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_45 2024-09-14T06:02:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:02:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:02:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_45 2024-09-14T06:02:23.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:23.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:23.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_45 2024-09-14T06:02:23.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2024-09-14T06:02:23.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_45: (2) No such file or directory 2024-09-14T06:02:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:02:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:02:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:02:23.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_45 deleted '' 2024-09-14T06:02:23.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:23.990 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_45 2024-09-14T06:02:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:02:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:02:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:02:23.991 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_45 2024-09-14T06:02:23.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:23.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:23.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_45 2024-09-14T06:02:23.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2024-09-14T06:02:23.992 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:24.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_45: (2) No such file or directory 2024-09-14T06:02:24.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:02:24.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:24.040 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:02:24.040 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:24.040 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:02:24.040 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_46 2024-09-14T06:02:24.040 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_46 deleted 2024-09-14T06:02:24.041 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:24.041 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:24.041 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_46 2024-09-14T06:02:24.041 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:02:24.041 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:24.041 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:02:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_46 2024-09-14T06:02:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_46 2024-09-14T06:02:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2024-09-14T06:02:24.043 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:24.078 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_46: (2) No such file or directory 2024-09-14T06:02:24.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:02:24.081 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:02:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:02:24.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_46 deleted '' 2024-09-14T06:02:24.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:24.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:24.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_46 2024-09-14T06:02:24.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:02:24.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:24.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:02:24.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:02:24.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_46 2024-09-14T06:02:24.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:24.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:24.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_46 2024-09-14T06:02:24.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2024-09-14T06:02:24.183 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:24.218 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_46: (2) No such file or directory 2024-09-14T06:02:24.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:02:24.222 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:24.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:02:24.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:24.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:02:24.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_47 2024-09-14T06:02:24.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_47 deleted 2024-09-14T06:02:24.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:24.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:24.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_47 2024-09-14T06:02:24.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:02:24.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:24.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:02:24.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:24.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_47 2024-09-14T06:02:24.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:24.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:24.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_47 2024-09-14T06:02:24.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2024-09-14T06:02:24.224 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:24.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_47: (2) No such file or directory 2024-09-14T06:02:24.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:02:24.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:02:24.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:02:24.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_47 deleted '' 2024-09-14T06:02:24.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:24.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:24.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_47 2024-09-14T06:02:24.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:02:24.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:24.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:02:24.363 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:02:24.363 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_47 2024-09-14T06:02:24.363 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:24.363 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:24.363 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_47 2024-09-14T06:02:24.363 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2024-09-14T06:02:24.363 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:24.399 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_47: (2) No such file or directory 2024-09-14T06:02:24.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:02:24.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:24.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:02:24.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:24.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:02:24.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_48 2024-09-14T06:02:24.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_48 deleted 2024-09-14T06:02:24.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:24.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:24.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_48 2024-09-14T06:02:24.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:02:24.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:24.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:02:24.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:24.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_48 2024-09-14T06:02:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_48 2024-09-14T06:02:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2024-09-14T06:02:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:24.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_48: (2) No such file or directory 2024-09-14T06:02:24.441 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:02:24.441 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:02:24.441 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:02:24.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_48 deleted '' 2024-09-14T06:02:24.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:24.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:24.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_48 2024-09-14T06:02:24.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:02:24.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:24.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:02:24.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:02:24.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_48 2024-09-14T06:02:24.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:24.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:24.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_48 2024-09-14T06:02:24.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2024-09-14T06:02:24.544 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_48: (2) No such file or directory 2024-09-14T06:02:24.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:02:24.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:24.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:02:24.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:24.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:02:24.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_49 2024-09-14T06:02:24.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_49 deleted 2024-09-14T06:02:24.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:24.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:24.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_49 2024-09-14T06:02:24.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:02:24.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:24.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:02:24.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:24.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_49 2024-09-14T06:02:24.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:24.584 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:24.584 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_49 2024-09-14T06:02:24.584 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2024-09-14T06:02:24.584 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:24.617 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_49: (2) No such file or directory 2024-09-14T06:02:24.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:02:24.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:02:24.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:02:24.722 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_49 deleted '' 2024-09-14T06:02:24.722 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:24.722 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:24.722 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_49 2024-09-14T06:02:24.722 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:02:24.722 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:24.722 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:02:24.722 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:02:24.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_49 2024-09-14T06:02:24.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:24.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:24.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_49 2024-09-14T06:02:24.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2024-09-14T06:02:24.723 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:24.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_49: (2) No such file or directory 2024-09-14T06:02:24.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:02:24.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:24.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:02:24.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:24.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-14T06:02:24.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_50 2024-09-14T06:02:24.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_50 deleted 2024-09-14T06:02:24.763 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:24.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:24.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_50 2024-09-14T06:02:24.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:02:24.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:24.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:02:24.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:24.765 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_50 2024-09-14T06:02:24.765 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:24.765 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:24.765 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_50 2024-09-14T06:02:24.765 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2024-09-14T06:02:24.765 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:24.796 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_50: (2) No such file or directory 2024-09-14T06:02:24.799 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:02:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:02:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:02:24.901 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_50 deleted '' 2024-09-14T06:02:24.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:24.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:24.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_50 2024-09-14T06:02:24.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:02:24.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:24.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:02:24.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:02:24.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_50 2024-09-14T06:02:24.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:24.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:24.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_50 2024-09-14T06:02:24.902 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2024-09-14T06:02:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:24.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_50: (2) No such file or directory 2024-09-14T06:02:24.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:02:24.940 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:02:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ testlog 'TEST: delete images during bootstrap' 2024-09-14T06:02:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ tee -a /tmp/tmp.sXArEbIrwv/rbd-mirror.test.log 2024-09-14T06:02:24.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ date '+%F %T' 2024-09-14T06:02:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo 2024-09-14 06:02:24 TEST: delete images during bootstrap 2024-09-14T06:02:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:2024-09-14 06:02:24 TEST: delete images during bootstrap 2024-09-14T06:02:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set_pool_mirror_mode cluster1 mirror image 2024-09-14T06:02:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:24.942 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=image 2024-09-14T06:02:24.943 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster1 mirror pool enable mirror image 2024-09-14T06:02:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd: mirroring is already configured for image mode 2024-09-14T06:02:24.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set_pool_mirror_mode cluster2 mirror image 2024-09-14T06:02:24.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:02:24.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:24.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=image 2024-09-14T06:02:24.980 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror image 2024-09-14T06:02:25.014 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd: mirroring is already configured for image mode 2024-09-14T06:02:25.020 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ start_mirror cluster1 2024-09-14T06:02:25.020 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:25.020 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local instance 2024-09-14T06:02:25.020 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set_cluster_instance cluster1 cluster instance 2024-09-14T06:02:25.020 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local val=cluster1 2024-09-14T06:02:25.021 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster_var_name=cluster 2024-09-14T06:02:25.021 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local instance_var_name=instance 2024-09-14T06:02:25.021 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cluster=cluster1 2024-09-14T06:02:25.021 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ instance=cluster1 2024-09-14T06:02:25.021 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' cluster1 = cluster1 ']' 2024-09-14T06:02:25.021 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ instance=0 2024-09-14T06:02:25.021 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ eval cluster=cluster1 2024-09-14T06:02:25.021 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ cluster=cluster1 2024-09-14T06:02:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ eval instance=0 2024-09-14T06:02:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ instance=0 2024-09-14T06:02:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 1 2024-09-14T06:02:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 2024-09-14T06:02:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=test 2024-09-14T06:02:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ seq 1 10 2024-09-14T06:02:25.022 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-14T06:02:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_1 2024-09-14T06:02:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_1 512M 2024-09-14T06:02:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:02:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T06:02:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T06:02:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-14T06:02:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T06:02:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T06:02:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 512M ']' 2024-09-14T06:02:25.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=512M 2024-09-14T06:02:25.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T06:02:25.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_1 2024-09-14T06:02:25.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_1 2024-09-14T06:02:25.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:02:25.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:25.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-14T06:02:25.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T06:02:25.072 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2024-09-14T06:02:25.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T06:02:25.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2024-09-14T06:02:25.230 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_1': 2024-09-14T06:02:25.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 512 MiB in 128 objects 2024-09-14T06:02:25.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T06:02:25.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T06:02:25.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 194cccdbaf50 2024-09-14T06:02:25.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.194cccdbaf50 2024-09-14T06:02:25.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T06:02:25.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T06:02:25.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T06:02:25.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T06:02:25.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 06:02:25 2024 2024-09-14T06:02:25.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 06:02:25 2024 2024-09-14T06:02:25.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 06:02:25 2024 2024-09-14T06:02:25.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T06:02:25.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T06:02:25.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 20b811eb-0cde-4293-8b47-9c30bf1fd281 2024-09-14T06:02:25.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T06:02:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_1 2024-09-14T06:02:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:02:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-14T06:02:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:02:25.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=9 2024-09-14T06:02:25.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T06:02:25.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:02:25.463 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 15546368 bytes 2024-09-14T06:02:25.594 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 39585792 bytes 2024-09-14T06:02:25.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 69219328 bytes 2024-09-14T06:02:25.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 102004736 bytes 2024-09-14T06:02:25.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 145098752 bytes 2024-09-14T06:02:26.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 173117440 bytes 2024-09-14T06:02:26.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 232510464 bytes 2024-09-14T06:02:26.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 233115648 bytes 2024-09-14T06:02:26.382 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 265886720 bytes 2024-09-14T06:02:26.498 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 309085184 bytes 2024-09-14T06:02:26.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 344479744 bytes 2024-09-14T06:02:26.725 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 369152000 bytes 2024-09-14T06:02:26.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 380682240 bytes 2024-09-14T06:02:26.963 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 442194944 bytes 2024-09-14T06:02:27.089 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 479890432 bytes 2024-09-14T06:02:27.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 495233024 bytes 2024-09-14T06:02:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 512570368 bytes 2024-09-14T06:02:27.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 539983872 bytes 2024-09-14T06:02:27.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 540591104 bytes 2024-09-14T06:02:27.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 592707584 bytes 2024-09-14T06:02:27.724 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 626121728 bytes 2024-09-14T06:02:27.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 676811776 bytes 2024-09-14T06:02:27.966 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 705313792 bytes 2024-09-14T06:02:28.047 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 705924096 bytes 2024-09-14T06:02:28.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 733504512 bytes 2024-09-14T06:02:28.288 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 798528512 bytes 2024-09-14T06:02:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 877028352 bytes 2024-09-14T06:02:28.538 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 887193600 bytes 2024-09-14T06:02:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 916659200 bytes 2024-09-14T06:02:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 993287168 bytes 2024-09-14T06:02:28.957 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1063758848 bytes 2024-09-14T06:02:29.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1074472960 bytes 2024-09-14T06:02:29.169 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1114492928 bytes 2024-09-14T06:02:29.293 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1172564992 bytes 2024-09-14T06:02:29.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1186531328 bytes 2024-09-14T06:02:29.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1194206208 bytes 2024-09-14T06:02:29.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1230915584 bytes 2024-09-14T06:02:29.685 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1235902464 bytes 2024-09-14T06:02:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1282208768 bytes 2024-09-14T06:02:29.954 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1337718784 bytes 2024-09-14T06:02:30.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1358458880 bytes 2024-09-14T06:02:30.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1387618304 bytes 2024-09-14T06:02:30.302 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1448443904 bytes 2024-09-14T06:02:30.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1492300800 bytes 2024-09-14T06:02:30.577 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1560105984 bytes 2024-09-14T06:02:30.681 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1582746624 bytes 2024-09-14T06:02:30.787 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 1609640960 bytes 2024-09-14T06:02:30.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 1626932224 bytes 2024-09-14T06:02:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 1627538432 bytes 2024-09-14T06:02:31.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 1757278208 bytes 2024-09-14T06:02:31.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 1775076352 bytes 2024-09-14T06:02:31.397 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 1819196416 bytes 2024-09-14T06:02:31.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 1874768896 bytes 2024-09-14T06:02:31.660 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 1925499904 bytes 2024-09-14T06:02:31.796 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 1991732224 bytes 2024-09-14T06:02:31.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2000627712 bytes 2024-09-14T06:02:32.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2124770304 bytes 2024-09-14T06:02:32.229 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2165752832 bytes 2024-09-14T06:02:32.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2171357184 bytes 2024-09-14T06:02:32.402 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2171984896 bytes 2024-09-14T06:02:32.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2191805440 bytes 2024-09-14T06:02:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2250184704 bytes 2024-09-14T06:02:32.789 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2280708096 bytes 2024-09-14T06:02:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2336026624 bytes 2024-09-14T06:02:33.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2392356864 bytes 2024-09-14T06:02:33.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 2398889984 bytes 2024-09-14T06:02:33.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 2445912064 bytes 2024-09-14T06:02:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 2518460416 bytes 2024-09-14T06:02:33.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 2592813056 bytes 2024-09-14T06:02:33.718 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 2631257088 bytes 2024-09-14T06:02:33.830 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 2651841536 bytes 2024-09-14T06:02:33.932 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 2674214912 bytes 2024-09-14T06:02:34.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7300 IOs, 2752934912 bytes 2024-09-14T06:02:34.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7400 IOs, 2794958848 bytes 2024-09-14T06:02:34.245 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T06:02:34.245 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T06:02:34.246 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:02:34.246 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:34.246 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_1 present 2024-09-14T06:02:34.246 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:34.246 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:34.246 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-14T06:02:34.246 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=present 2024-09-14T06:02:34.246 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:02:34.246 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:02:34.246 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:02:34.246 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_1 2024-09-14T06:02:34.246 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:34.246 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:34.246 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_1 2024-09-14T06:02:34.247 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-14T06:02:34.247 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:34.563 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id=127e11e9ce9d 2024-09-14T06:02:34.563 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:02:34.563 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:02:34.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_1 present 127e11e9ce9d 2024-09-14T06:02:34.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:02:34.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:34.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-14T06:02:34.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=present 2024-09-14T06:02:34.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id=127e11e9ce9d 2024-09-14T06:02:34.665 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:02:34.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:02:34.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_1 2024-09-14T06:02:34.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:02:34.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:02:34.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_1 2024-09-14T06:02:34.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-14T06:02:34.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:02:34.714 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id=127e11e9ce9d 2024-09-14T06:02:34.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 127e11e9ce9d 2024-09-14T06:02:34.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -z 127e11e9ce9d -o 127e11e9ce9d = 127e11e9ce9d 2024-09-14T06:02:34.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=present 2024-09-14T06:02:34.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test present = present 2024-09-14T06:02:34.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:02:34.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ disable_mirror cluster2 mirror image_1 2024-09-14T06:02:34.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:02:34.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:02:34.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-14T06:02:34.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_1 2024-09-14T06:02:47.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:02:47.724+0000 7f1bc52a6640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55710c4fab60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T06:02:47.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:02:47.724+0000 7f1bcb2b2640 -1 rbd::mirror::ImageReplayer: 0x55710cdb8a00 [3/20b811eb-0cde-4293-8b47-9c30bf1fd281] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T06:03:05.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-14T06:03:05.221+0000 7f9b14727640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55f03b575000 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=185 2024-09-14T06:03:05.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-14T06:03:05.221+0000 7f9b14727640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55f03b575000 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=189, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=9d38de88-cffd-4132-b7e8-809c69f2745a, primary_snap_id=b9, last_copied_object_number=128, snap_seqs={185=18446744073709551614}] 2024-09-14T06:03:05.231 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring disabled 2024-09-14T06:03:05.264 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2024-09-14T06:03:05.264 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:03:05.264 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:03:05.264 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-14T06:03:05.264 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:03:05.264 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:03:05.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:03:05.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:03:05.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_1 2024-09-14T06:03:05.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:03:05.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:03:05.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_1 2024-09-14T06:03:05.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-14T06:03:05.265 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:03:05.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-14T06:03:05.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:03:05.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:03:05.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:03:05.408 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2024-09-14T06:03:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:03:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:03:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-14T06:03:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:03:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:03:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:03:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:03:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_1 2024-09-14T06:03:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:03:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:03:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_1 2024-09-14T06:03:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-14T06:03:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:03:05.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-14T06:03:05.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:03:05.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:03:05.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:03:05.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:03:05.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ purge_snapshots cluster2 mirror image_1 2024-09-14T06:03:05.450 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:03:05.450 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:03:05.450 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-14T06:03:05.450 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap purge mirror/image_1 2024-09-14T06:03:05.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_1 2024-09-14T06:03:05.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:03:05.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:03:05.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-14T06:03:05.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:03:05.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:03:05.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_1 2024-09-14T06:03:05.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:03:05.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:03:05.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_1 2024-09-14T06:03:05.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-09-14T06:03:05.542 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-09-14T06:03:05.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-14T06:03:05.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:03:05.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-14T06:03:05.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_2 2024-09-14T06:03:05.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_2 512M 2024-09-14T06:03:05.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:03:05.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T06:03:05.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:03:05.791 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T06:03:05.791 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-14T06:03:05.791 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T06:03:05.791 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T06:03:05.791 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 512M ']' 2024-09-14T06:03:05.791 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=512M 2024-09-14T06:03:05.791 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T06:03:05.791 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_2 2024-09-14T06:03:05.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_2 2024-09-14T06:03:05.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:03:05.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:03:05.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-14T06:03:05.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T06:03:05.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2024-09-14T06:03:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T06:03:06.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2024-09-14T06:03:06.296 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_2': 2024-09-14T06:03:06.296 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 512 MiB in 128 objects 2024-09-14T06:03:06.296 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T06:03:06.296 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T06:03:06.296 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 196442ed59dc 2024-09-14T06:03:06.296 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.196442ed59dc 2024-09-14T06:03:06.296 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T06:03:06.296 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T06:03:06.296 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T06:03:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T06:03:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 06:03:05 2024 2024-09-14T06:03:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 06:03:05 2024 2024-09-14T06:03:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 06:03:05 2024 2024-09-14T06:03:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T06:03:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T06:03:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 126f773c-d6e9-4e21-957f-3b03392a9a37 2024-09-14T06:03:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T06:03:06.299 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_2 2024-09-14T06:03:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:03:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:03:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-14T06:03:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:03:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=7 2024-09-14T06:03:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T06:03:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:03:06.791 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 57675776 bytes 2024-09-14T06:03:06.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 67932160 bytes 2024-09-14T06:03:07.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 129193984 bytes 2024-09-14T06:03:07.211 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 206465024 bytes 2024-09-14T06:03:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 238419968 bytes 2024-09-14T06:03:07.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 319344640 bytes 2024-09-14T06:03:07.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 343187456 bytes 2024-09-14T06:03:07.680 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 359172096 bytes 2024-09-14T06:03:07.790 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 388461568 bytes 2024-09-14T06:03:07.908 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 416280576 bytes 2024-09-14T06:03:08.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 474550272 bytes 2024-09-14T06:03:08.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 520153088 bytes 2024-09-14T06:03:08.251 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 524647424 bytes 2024-09-14T06:03:08.375 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 569634816 bytes 2024-09-14T06:03:08.588 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 711822336 bytes 2024-09-14T06:03:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 752996352 bytes 2024-09-14T06:03:08.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 786299904 bytes 2024-09-14T06:03:08.951 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 822456320 bytes 2024-09-14T06:03:09.076 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 866868224 bytes 2024-09-14T06:03:09.225 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 941215744 bytes 2024-09-14T06:03:09.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 996777984 bytes 2024-09-14T06:03:09.479 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 1038941184 bytes 2024-09-14T06:03:09.613 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1095821312 bytes 2024-09-14T06:03:09.704 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1107359744 bytes 2024-09-14T06:03:09.728 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-14T06:03:09.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-14T06:03:09.729 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-09-14T06:03:09.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-14T06:03:09.729+0000 7f43081dd640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 65518) UID: 1000 2024-09-14T06:03:09.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1160812544 bytes 2024-09-14T06:03:09.984 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1222821888 bytes 2024-09-14T06:03:10.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1233300480 bytes 2024-09-14T06:03:10.181 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1261393920 bytes 2024-09-14T06:03:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1305363456 bytes 2024-09-14T06:03:10.464 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1379621888 bytes 2024-09-14T06:03:10.570 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1411761152 bytes 2024-09-14T06:03:10.689 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1466301440 bytes 2024-09-14T06:03:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1517159424 bytes 2024-09-14T06:03:10.924 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1517755392 bytes 2024-09-14T06:03:11.024 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1536913408 bytes 2024-09-14T06:03:11.189 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1619150848 bytes 2024-09-14T06:03:11.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1665531904 bytes 2024-09-14T06:03:11.412 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1693843456 bytes 2024-09-14T06:03:11.528 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1720273920 bytes 2024-09-14T06:03:11.635 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1748482048 bytes 2024-09-14T06:03:11.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1749146624 bytes 2024-09-14T06:03:11.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1749735424 bytes 2024-09-14T06:03:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1760023552 bytes 2024-09-14T06:03:12.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1817793536 bytes 2024-09-14T06:03:12.226 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1931944960 bytes 2024-09-14T06:03:12.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1988041728 bytes 2024-09-14T06:03:12.468 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 2018101248 bytes 2024-09-14T06:03:12.607 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2085181440 bytes 2024-09-14T06:03:12.750 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2134337536 bytes 2024-09-14T06:03:12.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2156547072 bytes 2024-09-14T06:03:12.996 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2217867264 bytes 2024-09-14T06:03:13.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2265418752 bytes 2024-09-14T06:03:13.221 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2266038272 bytes 2024-09-14T06:03:13.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T06:03:13.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T06:03:13.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:03:13.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:03:13.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_2 present 2024-09-14T06:03:13.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:03:13.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:03:13.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-14T06:03:13.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=present 2024-09-14T06:03:13.308 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:03:13.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:03:13.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:03:13.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_2 2024-09-14T06:03:13.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:03:13.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:03:13.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_2 2024-09-14T06:03:13.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-14T06:03:13.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:03:13.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id=127ef92b28c1 2024-09-14T06:03:13.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:03:13.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:03:13.454 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_2 present 127ef92b28c1 2024-09-14T06:03:13.454 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:03:13.454 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:03:13.454 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-14T06:03:13.454 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=present 2024-09-14T06:03:13.454 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id=127ef92b28c1 2024-09-14T06:03:13.455 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:03:13.455 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:03:13.455 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_2 2024-09-14T06:03:13.455 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:03:13.455 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:03:13.455 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_2 2024-09-14T06:03:13.455 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-14T06:03:13.455 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:03:13.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id=127ef92b28c1 2024-09-14T06:03:13.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 127ef92b28c1 2024-09-14T06:03:13.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -z 127ef92b28c1 -o 127ef92b28c1 = 127ef92b28c1 2024-09-14T06:03:13.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=present 2024-09-14T06:03:13.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test present = present 2024-09-14T06:03:13.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:03:13.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ disable_mirror cluster2 mirror image_2 2024-09-14T06:03:13.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:03:13.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:03:13.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-14T06:03:13.501 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_2 2024-09-14T06:03:16.731 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-14T06:03:16.731 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T06:03:16.731 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-14T06:03:16.731 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-14T06:03:16.731 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-14T06:03:16.735 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-14T06:03:16.735 INFO:tasks.rbd_mirror_thrash:waiting for 115 secs before thrashing 2024-09-14T06:03:17.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:03:17.723+0000 7f1bc4aa5640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55710ca864e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T06:03:17.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:03:17.723+0000 7f1bcb2b2640 -1 rbd::mirror::ImageReplayer: 0x55710cdb9b80 [3/126f773c-d6e9-4e21-957f-3b03392a9a37] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T06:03:24.634 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-14T06:03:24.634 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-14T06:03:24.635 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-09-14T06:03:24.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-14T06:03:24.634+0000 7f1965236640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 57686) UID: 1000 2024-09-14T06:03:30.636 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-14T06:03:30.636 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T06:03:30.636 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-14T06:03:30.636 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-14T06:03:30.637 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-14T06:03:30.640 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-14T06:03:30.640 INFO:tasks.rbd_mirror_thrash:waiting for 86 secs before thrashing 2024-09-14T06:03:44.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-14T06:03:44.292+0000 7f9b14f28640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55f03bf27800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=186 2024-09-14T06:03:44.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-14T06:03:44.292+0000 7f9b14f28640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55f03bf27800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=190, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=9d38de88-cffd-4132-b7e8-809c69f2745a, primary_snap_id=ba, last_copied_object_number=128, snap_seqs={186=18446744073709551614}] 2024-09-14T06:03:44.299 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring disabled 2024-09-14T06:03:44.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2024-09-14T06:03:44.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:03:44.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:03:44.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-14T06:03:44.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:03:44.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:03:44.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:03:44.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:03:44.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_2 2024-09-14T06:03:44.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:03:44.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:03:44.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_2 2024-09-14T06:03:44.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-14T06:03:44.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:03:44.360 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-14T06:03:44.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:03:44.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:03:44.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:03:44.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2024-09-14T06:03:44.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:03:44.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:03:44.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-14T06:03:44.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:03:44.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:03:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:03:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:03:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_2 2024-09-14T06:03:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:03:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:03:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_2 2024-09-14T06:03:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-14T06:03:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:03:44.503 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-14T06:03:44.506 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:03:44.506 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:03:44.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:03:44.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:03:44.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ purge_snapshots cluster2 mirror image_2 2024-09-14T06:03:44.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:03:44.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:03:44.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-14T06:03:44.507 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap purge mirror/image_2 2024-09-14T06:03:44.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_2 2024-09-14T06:03:44.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:03:44.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:03:44.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-14T06:03:44.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:03:44.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:03:44.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_2 2024-09-14T06:03:44.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:03:44.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:03:44.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_2 2024-09-14T06:03:44.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-09-14T06:03:44.801 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-09-14T06:03:45.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-14T06:03:45.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:03:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-14T06:03:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_3 2024-09-14T06:03:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_3 512M 2024-09-14T06:03:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:03:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T06:03:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:03:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T06:03:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-14T06:03:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T06:03:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T06:03:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 512M ']' 2024-09-14T06:03:45.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=512M 2024-09-14T06:03:45.054 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T06:03:45.054 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_3 2024-09-14T06:03:45.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_3 2024-09-14T06:03:45.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:03:45.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:03:45.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-14T06:03:45.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T06:03:45.106 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2024-09-14T06:03:46.303 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T06:03:46.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2024-09-14T06:03:46.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_3': 2024-09-14T06:03:46.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 512 MiB in 128 objects 2024-09-14T06:03:46.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T06:03:46.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T06:03:46.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 198c5a011fe9 2024-09-14T06:03:46.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.198c5a011fe9 2024-09-14T06:03:46.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T06:03:46.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T06:03:46.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T06:03:46.358 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T06:03:46.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 06:03:45 2024 2024-09-14T06:03:46.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 06:03:45 2024 2024-09-14T06:03:46.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 06:03:45 2024 2024-09-14T06:03:46.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T06:03:46.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T06:03:46.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 93adcf91-0871-4ef5-b987-df3d49c5cc28 2024-09-14T06:03:46.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T06:03:46.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_3 2024-09-14T06:03:46.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:03:46.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:03:46.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-14T06:03:46.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:03:46.363 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=7 2024-09-14T06:03:46.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T06:03:46.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:03:46.602 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 18780160 bytes 2024-09-14T06:03:46.744 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 55918592 bytes 2024-09-14T06:03:46.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 72962048 bytes 2024-09-14T06:03:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 144389120 bytes 2024-09-14T06:03:47.163 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 220856320 bytes 2024-09-14T06:03:47.291 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 263957504 bytes 2024-09-14T06:03:47.400 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 287271936 bytes 2024-09-14T06:03:47.562 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 374133760 bytes 2024-09-14T06:03:47.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 410817536 bytes 2024-09-14T06:03:47.816 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 473013248 bytes 2024-09-14T06:03:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 494565376 bytes 2024-09-14T06:03:48.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 527824896 bytes 2024-09-14T06:03:48.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 567497728 bytes 2024-09-14T06:03:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 588976128 bytes 2024-09-14T06:03:48.586 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 650149888 bytes 2024-09-14T06:03:48.698 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 680681472 bytes 2024-09-14T06:03:48.857 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 758106112 bytes 2024-09-14T06:03:48.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 758756352 bytes 2024-09-14T06:03:49.070 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 804664320 bytes 2024-09-14T06:03:49.182 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 833444864 bytes 2024-09-14T06:03:49.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 892235776 bytes 2024-09-14T06:03:49.473 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 970812416 bytes 2024-09-14T06:03:49.574 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 988821504 bytes 2024-09-14T06:03:49.715 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1059912704 bytes 2024-09-14T06:03:49.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1104782336 bytes 2024-09-14T06:03:49.993 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1178582016 bytes 2024-09-14T06:03:50.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1197349888 bytes 2024-09-14T06:03:50.228 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1252993024 bytes 2024-09-14T06:03:50.337 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1285104640 bytes 2024-09-14T06:03:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1292744704 bytes 2024-09-14T06:03:50.563 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1342390272 bytes 2024-09-14T06:03:50.657 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1351864320 bytes 2024-09-14T06:03:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1412290560 bytes 2024-09-14T06:03:50.914 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1451574272 bytes 2024-09-14T06:03:51.020 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1473249280 bytes 2024-09-14T06:03:51.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1514080256 bytes 2024-09-14T06:03:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1583080448 bytes 2024-09-14T06:03:51.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1615010816 bytes 2024-09-14T06:03:51.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1665931264 bytes 2024-09-14T06:03:51.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1724079104 bytes 2024-09-14T06:03:51.776 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1744840704 bytes 2024-09-14T06:03:51.904 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1792212992 bytes 2024-09-14T06:03:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1805009920 bytes 2024-09-14T06:03:52.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1869724672 bytes 2024-09-14T06:03:52.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1930442752 bytes 2024-09-14T06:03:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 2003466240 bytes 2024-09-14T06:03:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 2056559616 bytes 2024-09-14T06:03:52.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2080354304 bytes 2024-09-14T06:03:52.795 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2124806144 bytes 2024-09-14T06:03:52.962 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2210361344 bytes 2024-09-14T06:03:53.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2322425856 bytes 2024-09-14T06:03:53.271 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2352929792 bytes 2024-09-14T06:03:53.371 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T06:03:53.371 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T06:03:53.371 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:03:53.371 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:03:53.371 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_3 present 2024-09-14T06:03:53.371 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:03:53.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:03:53.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-14T06:03:53.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=present 2024-09-14T06:03:53.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:03:53.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:03:53.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:03:53.373 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_3 2024-09-14T06:03:53.373 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:03:53.373 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:03:53.373 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_3 2024-09-14T06:03:53.373 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-14T06:03:53.373 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:03:53.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id=127e939d1bb 2024-09-14T06:03:53.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:03:53.490 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:03:53.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_3 present 127e939d1bb 2024-09-14T06:03:53.591 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:03:53.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:03:53.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-14T06:03:53.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=present 2024-09-14T06:03:53.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id=127e939d1bb 2024-09-14T06:03:53.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:03:53.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:03:53.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_3 2024-09-14T06:03:53.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:03:53.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:03:53.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_3 2024-09-14T06:03:53.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-14T06:03:53.592 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:03:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id=127e939d1bb 2024-09-14T06:03:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 127e939d1bb 2024-09-14T06:03:53.690 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -z 127e939d1bb -o 127e939d1bb = 127e939d1bb 2024-09-14T06:03:53.690 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=present 2024-09-14T06:03:53.690 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test present = present 2024-09-14T06:03:53.690 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:03:53.690 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ disable_mirror cluster2 mirror image_3 2024-09-14T06:03:53.690 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:03:53.690 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:03:53.690 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-14T06:03:53.690 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_3 2024-09-14T06:04:17.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:04:17.726+0000 7f1bc52a6640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55710d3036c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T06:04:17.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:04:17.726+0000 7f1bcb2b2640 -1 rbd::mirror::ImageReplayer: 0x55710cdb9b80 [3/93adcf91-0871-4ef5-b987-df3d49c5cc28] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T06:04:23.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-14T06:04:23.202+0000 7f9b14f28640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55f03a7d2c00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=187 2024-09-14T06:04:23.204 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-14T06:04:23.202+0000 7f9b14f28640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55f03a7d2c00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=191, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=9d38de88-cffd-4132-b7e8-809c69f2745a, primary_snap_id=bb, last_copied_object_number=128, snap_seqs={187=18446744073709551614}] 2024-09-14T06:04:23.221 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring disabled 2024-09-14T06:04:23.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2024-09-14T06:04:23.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:04:23.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:04:23.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-14T06:04:23.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:04:23.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:04:23.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:04:23.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:04:23.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_3 2024-09-14T06:04:23.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:04:23.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:04:23.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_3 2024-09-14T06:04:23.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-14T06:04:23.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:04:23.268 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:04:23.266+0000 7f1bcb2b2640 -1 rbd::mirror::ImageReplayer: 0x55710cdb9b80 [3/93adcf91-0871-4ef5-b987-df3d49c5cc28] start: already running 2024-09-14T06:04:23.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-14T06:04:23.287 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:04:23.287 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:04:23.287 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:04:23.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2024-09-14T06:04:23.389 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:04:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:04:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-14T06:04:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:04:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:04:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:04:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:04:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_3 2024-09-14T06:04:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:04:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:04:23.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_3 2024-09-14T06:04:23.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-14T06:04:23.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:04:23.426 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-14T06:04:23.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:04:23.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:04:23.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:04:23.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:04:23.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ purge_snapshots cluster2 mirror image_3 2024-09-14T06:04:23.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:04:23.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:04:23.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-14T06:04:23.430 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap purge mirror/image_3 2024-09-14T06:04:23.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_3 2024-09-14T06:04:23.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:04:23.477 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:04:23.477 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-14T06:04:23.477 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:04:23.477 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:04:23.478 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_3 2024-09-14T06:04:23.478 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:04:23.478 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:04:23.478 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_3 2024-09-14T06:04:23.478 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-09-14T06:04:23.524 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-09-14T06:04:23.770 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-14T06:04:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:04:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-14T06:04:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_4 2024-09-14T06:04:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_4 512M 2024-09-14T06:04:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:04:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T06:04:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:04:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T06:04:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-14T06:04:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T06:04:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T06:04:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 512M ']' 2024-09-14T06:04:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=512M 2024-09-14T06:04:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T06:04:23.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_4 2024-09-14T06:04:23.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_4 2024-09-14T06:04:23.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:04:23.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:04:23.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-14T06:04:23.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T06:04:23.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2024-09-14T06:04:24.220 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T06:04:24.233 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2024-09-14T06:04:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_4': 2024-09-14T06:04:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 512 MiB in 128 objects 2024-09-14T06:04:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T06:04:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T06:04:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 19a4b44c6559 2024-09-14T06:04:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.19a4b44c6559 2024-09-14T06:04:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T06:04:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T06:04:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T06:04:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T06:04:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 06:04:23 2024 2024-09-14T06:04:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 06:04:23 2024 2024-09-14T06:04:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 06:04:23 2024 2024-09-14T06:04:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T06:04:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T06:04:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 49e409d8-6a04-4841-bfd2-d788b4bd003e 2024-09-14T06:04:24.275 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T06:04:24.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_4 2024-09-14T06:04:24.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:04:24.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:04:24.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-14T06:04:24.277 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:04:24.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=11 2024-09-14T06:04:24.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T06:04:24.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:04:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 45222912 bytes 2024-09-14T06:04:24.685 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 75701248 bytes 2024-09-14T06:04:24.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 132129792 bytes 2024-09-14T06:04:24.947 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 204664832 bytes 2024-09-14T06:04:25.116 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 277465088 bytes 2024-09-14T06:04:25.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 307415040 bytes 2024-09-14T06:04:25.311 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 308046848 bytes 2024-09-14T06:04:25.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 409070592 bytes 2024-09-14T06:04:25.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 445763584 bytes 2024-09-14T06:04:25.731 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 503831552 bytes 2024-09-14T06:04:25.855 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 550463488 bytes 2024-09-14T06:04:25.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 551059456 bytes 2024-09-14T06:04:26.050 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 592837632 bytes 2024-09-14T06:04:26.160 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 621921280 bytes 2024-09-14T06:04:26.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 663815168 bytes 2024-09-14T06:04:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 712420352 bytes 2024-09-14T06:04:26.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 745525248 bytes 2024-09-14T06:04:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 826486784 bytes 2024-09-14T06:04:26.811 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 893585408 bytes 2024-09-14T06:04:26.937 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 939693056 bytes 2024-09-14T06:04:27.036 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 968621056 bytes 2024-09-14T06:04:27.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 1003776000 bytes 2024-09-14T06:04:27.274 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1045494784 bytes 2024-09-14T06:04:27.382 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1074292736 bytes 2024-09-14T06:04:27.478 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1085981696 bytes 2024-09-14T06:04:27.595 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1121766400 bytes 2024-09-14T06:04:27.689 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1135733760 bytes 2024-09-14T06:04:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1201728512 bytes 2024-09-14T06:04:27.964 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1258238976 bytes 2024-09-14T06:04:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1263016960 bytes 2024-09-14T06:04:28.161 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1293702144 bytes 2024-09-14T06:04:28.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1310991360 bytes 2024-09-14T06:04:28.341 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1331874816 bytes 2024-09-14T06:04:28.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1381174272 bytes 2024-09-14T06:04:28.604 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1421742080 bytes 2024-09-14T06:04:28.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1457504256 bytes 2024-09-14T06:04:28.827 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1488595968 bytes 2024-09-14T06:04:28.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1555656704 bytes 2024-09-14T06:04:29.149 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1665324032 bytes 2024-09-14T06:04:29.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1689720832 bytes 2024-09-14T06:04:29.352 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1697932288 bytes 2024-09-14T06:04:29.509 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1780414464 bytes 2024-09-14T06:04:29.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1781699584 bytes 2024-09-14T06:04:29.699 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1806891008 bytes 2024-09-14T06:04:29.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1887634432 bytes 2024-09-14T06:04:29.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1920344064 bytes 2024-09-14T06:04:30.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 2038530048 bytes 2024-09-14T06:04:30.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2044579840 bytes 2024-09-14T06:04:30.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2106691584 bytes 2024-09-14T06:04:30.542 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2164898816 bytes 2024-09-14T06:04:30.646 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2192666624 bytes 2024-09-14T06:04:30.738 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2201494528 bytes 2024-09-14T06:04:30.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2340094976 bytes 2024-09-14T06:04:31.145 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2466570240 bytes 2024-09-14T06:04:31.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2468869120 bytes 2024-09-14T06:04:31.392 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2543184896 bytes 2024-09-14T06:04:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2591006720 bytes 2024-09-14T06:04:31.701 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2688441344 bytes 2024-09-14T06:04:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2730509312 bytes 2024-09-14T06:04:31.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2763260928 bytes 2024-09-14T06:04:32.062 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2806881280 bytes 2024-09-14T06:04:32.144 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2807503872 bytes 2024-09-14T06:04:32.271 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2852671488 bytes 2024-09-14T06:04:32.394 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2889916416 bytes 2024-09-14T06:04:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2913433600 bytes 2024-09-14T06:04:32.597 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 2926314496 bytes 2024-09-14T06:04:32.687 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 2953313280 bytes 2024-09-14T06:04:32.821 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 2991023104 bytes 2024-09-14T06:04:32.979 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 3071523840 bytes 2024-09-14T06:04:33.067 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 3074152448 bytes 2024-09-14T06:04:33.190 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 3104667648 bytes 2024-09-14T06:04:33.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 3182418944 bytes 2024-09-14T06:04:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7300 IOs, 3219119104 bytes 2024-09-14T06:04:33.609 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7400 IOs, 3285121024 bytes 2024-09-14T06:04:33.731 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7500 IOs, 3319979008 bytes 2024-09-14T06:04:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7600 IOs, 3371894784 bytes 2024-09-14T06:04:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7700 IOs, 3422242816 bytes 2024-09-14T06:04:34.096 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7800 IOs, 3455514624 bytes 2024-09-14T06:04:34.175 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7900 IOs, 3457268736 bytes 2024-09-14T06:04:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8000 IOs, 3458104320 bytes 2024-09-14T06:04:34.341 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8100 IOs, 3458700288 bytes 2024-09-14T06:04:34.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8200 IOs, 3482757120 bytes 2024-09-14T06:04:34.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8300 IOs, 3577935872 bytes 2024-09-14T06:04:34.747 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8400 IOs, 3615907840 bytes 2024-09-14T06:04:34.828 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8500 IOs, 3620732928 bytes 2024-09-14T06:04:34.916 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8600 IOs, 3625846784 bytes 2024-09-14T06:04:35.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8700 IOs, 3661892608 bytes 2024-09-14T06:04:35.171 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8800 IOs, 3722619904 bytes 2024-09-14T06:04:35.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T06:04:35.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T06:04:35.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:04:35.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:04:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_4 present 2024-09-14T06:04:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:04:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:04:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-14T06:04:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=present 2024-09-14T06:04:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:04:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:04:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:04:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_4 2024-09-14T06:04:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:04:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:04:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_4 2024-09-14T06:04:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-14T06:04:35.291 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:04:35.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id=127ec45b636b 2024-09-14T06:04:35.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:04:35.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:04:35.436 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_4 present 127ec45b636b 2024-09-14T06:04:35.436 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:04:35.436 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:04:35.436 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-14T06:04:35.436 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=present 2024-09-14T06:04:35.436 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id=127ec45b636b 2024-09-14T06:04:35.436 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:04:35.436 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:04:35.436 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_4 2024-09-14T06:04:35.437 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:04:35.437 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:04:35.437 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_4 2024-09-14T06:04:35.437 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-14T06:04:35.437 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:04:35.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id=127ec45b636b 2024-09-14T06:04:35.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 127ec45b636b 2024-09-14T06:04:35.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -z 127ec45b636b -o 127ec45b636b = 127ec45b636b 2024-09-14T06:04:35.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=present 2024-09-14T06:04:35.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test present = present 2024-09-14T06:04:35.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:04:35.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ disable_mirror cluster2 mirror image_4 2024-09-14T06:04:35.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:04:35.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:04:35.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-14T06:04:35.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_4 2024-09-14T06:04:47.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:04:47.727+0000 7f1bc4aa5640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55710cc87040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T06:04:47.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:04:47.727+0000 7f1bcbab3640 -1 rbd::mirror::ImageReplayer: 0x55710cdb9b80 [3/49e409d8-6a04-4841-bfd2-d788b4bd003e] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T06:04:56.641 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.48395540479854837) > weight (0.25) 2024-09-14T06:04:56.642 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.6154174716066846) > weight (0.25) 2024-09-14T06:04:56.642 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.739517354585705) > weight (0.25) 2024-09-14T06:04:56.642 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.817737809120711) > weight (0.25) 2024-09-14T06:04:56.642 INFO:tasks.rbd_mirror_thrash:waiting for 81 secs before thrashing 2024-09-14T06:05:06.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-14T06:05:06.260+0000 7f9b14727640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55f039b40400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=188 2024-09-14T06:05:06.261 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-14T06:05:06.260+0000 7f9b14727640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55f039b40400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=192, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=9d38de88-cffd-4132-b7e8-809c69f2745a, primary_snap_id=bc, last_copied_object_number=128, snap_seqs={188=18446744073709551614}] 2024-09-14T06:05:06.273 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring disabled 2024-09-14T06:05:06.296 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2024-09-14T06:05:06.296 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:05:06.296 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:05:06.296 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-14T06:05:06.296 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:05:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:05:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:05:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:05:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_4 2024-09-14T06:05:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:05:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:05:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_4 2024-09-14T06:05:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-14T06:05:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:05:06.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-14T06:05:06.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:05:06.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:05:06.339 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:05:06.441 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2024-09-14T06:05:06.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:05:06.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:05:06.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-14T06:05:06.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:05:06.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:05:06.442 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:05:06.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:05:06.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_4 2024-09-14T06:05:06.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:05:06.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:05:06.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_4 2024-09-14T06:05:06.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-14T06:05:06.443 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:05:06.477 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-14T06:05:06.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:05:06.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:05:06.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:05:06.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:05:06.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ purge_snapshots cluster2 mirror image_4 2024-09-14T06:05:06.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:05:06.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:05:06.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-14T06:05:06.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap purge mirror/image_4 2024-09-14T06:05:06.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_4 2024-09-14T06:05:06.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:05:06.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:05:06.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-14T06:05:06.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:05:06.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:05:06.528 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_4 2024-09-14T06:05:06.528 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:05:06.528 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:05:06.528 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_4 2024-09-14T06:05:06.528 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-09-14T06:05:06.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-09-14T06:05:06.825 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-14T06:05:06.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:05:06.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-14T06:05:06.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_5 2024-09-14T06:05:06.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_5 512M 2024-09-14T06:05:06.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:05:06.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T06:05:06.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:05:06.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T06:05:06.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-14T06:05:06.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T06:05:06.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T06:05:06.829 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 512M ']' 2024-09-14T06:05:06.830 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=512M 2024-09-14T06:05:06.830 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T06:05:06.830 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_5 2024-09-14T06:05:07.084 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_5 2024-09-14T06:05:07.084 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:05:07.085 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:05:07.085 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-14T06:05:07.085 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T06:05:07.085 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2024-09-14T06:05:07.266 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T06:05:07.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2024-09-14T06:05:07.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_5': 2024-09-14T06:05:07.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 512 MiB in 128 objects 2024-09-14T06:05:07.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T06:05:07.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T06:05:07.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 19bc4f55abe1 2024-09-14T06:05:07.316 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.19bc4f55abe1 2024-09-14T06:05:07.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T06:05:07.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T06:05:07.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T06:05:07.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T06:05:07.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 06:05:07 2024 2024-09-14T06:05:07.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 06:05:07 2024 2024-09-14T06:05:07.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 06:05:07 2024 2024-09-14T06:05:07.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T06:05:07.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T06:05:07.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: ce7dc856-80fa-4c8d-a257-16e1dd6680e8 2024-09-14T06:05:07.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T06:05:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_5 2024-09-14T06:05:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:05:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:05:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-14T06:05:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:05:07.322 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=8 2024-09-14T06:05:07.322 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T06:05:07.322 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:05:07.593 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 53422080 bytes 2024-09-14T06:05:07.764 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 122237952 bytes 2024-09-14T06:05:07.930 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 206542848 bytes 2024-09-14T06:05:08.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 267108352 bytes 2024-09-14T06:05:08.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 317731840 bytes 2024-09-14T06:05:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 343291904 bytes 2024-09-14T06:05:08.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 390501376 bytes 2024-09-14T06:05:08.559 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 444807168 bytes 2024-09-14T06:05:08.663 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 455432192 bytes 2024-09-14T06:05:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 514147328 bytes 2024-09-14T06:05:08.911 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 545725440 bytes 2024-09-14T06:05:09.007 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 554839040 bytes 2024-09-14T06:05:09.142 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 616044544 bytes 2024-09-14T06:05:09.341 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 735439872 bytes 2024-09-14T06:05:09.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 738750464 bytes 2024-09-14T06:05:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 758885376 bytes 2024-09-14T06:05:09.646 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 789414912 bytes 2024-09-14T06:05:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 801238016 bytes 2024-09-14T06:05:09.840 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 820813824 bytes 2024-09-14T06:05:10.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 910474240 bytes 2024-09-14T06:05:10.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 961140736 bytes 2024-09-14T06:05:10.243 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 986612736 bytes 2024-09-14T06:05:10.342 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1003851776 bytes 2024-09-14T06:05:10.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1078503424 bytes 2024-09-14T06:05:10.612 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1111163904 bytes 2024-09-14T06:05:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1141607424 bytes 2024-09-14T06:05:10.835 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1175052288 bytes 2024-09-14T06:05:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1192580096 bytes 2024-09-14T06:05:11.038 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1220973568 bytes 2024-09-14T06:05:11.180 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1274101760 bytes 2024-09-14T06:05:11.321 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1341092864 bytes 2024-09-14T06:05:11.473 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1413024768 bytes 2024-09-14T06:05:11.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1471418368 bytes 2024-09-14T06:05:11.724 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1492610048 bytes 2024-09-14T06:05:11.736 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9591329752532818) > weight (0.25) 2024-09-14T06:05:11.737 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-09-14T06:05:11.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-09-14T06:05:11.737 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-09-14T06:05:11.737 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-09-14T06:05:11.737 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T06:05:11.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:2024-09-14T06:05:11.737+0000 7f58f38b4640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 56857) UID: 1000 2024-09-14T06:05:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1563244544 bytes 2024-09-14T06:05:11.997 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1601746944 bytes 2024-09-14T06:05:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1629323264 bytes 2024-09-14T06:05:12.284 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1718372352 bytes 2024-09-14T06:05:12.407 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1762184192 bytes 2024-09-14T06:05:12.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1814805504 bytes 2024-09-14T06:05:12.709 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1906644992 bytes 2024-09-14T06:05:12.875 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1992229888 bytes 2024-09-14T06:05:13.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 2096115712 bytes 2024-09-14T06:05:13.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 2136402944 bytes 2024-09-14T06:05:13.299 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 2159711232 bytes 2024-09-14T06:05:13.426 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 2205666304 bytes 2024-09-14T06:05:13.540 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 2240735232 bytes 2024-09-14T06:05:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2287946752 bytes 2024-09-14T06:05:13.872 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2414955520 bytes 2024-09-14T06:05:13.998 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2456681472 bytes 2024-09-14T06:05:14.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2472221696 bytes 2024-09-14T06:05:14.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2540732416 bytes 2024-09-14T06:05:14.347 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2581815296 bytes 2024-09-14T06:05:14.532 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2681188352 bytes 2024-09-14T06:05:14.619 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2691997696 bytes 2024-09-14T06:05:14.724 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2712908800 bytes 2024-09-14T06:05:14.850 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2757182464 bytes 2024-09-14T06:05:14.960 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2779761664 bytes 2024-09-14T06:05:15.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2793966592 bytes 2024-09-14T06:05:15.139 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2794986496 bytes 2024-09-14T06:05:15.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2818988032 bytes 2024-09-14T06:05:15.328 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T06:05:15.328 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T06:05:15.328 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:05:15.328 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:05:15.328 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_5 present 2024-09-14T06:05:15.328 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:05:15.328 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:05:15.328 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-14T06:05:15.328 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=present 2024-09-14T06:05:15.328 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:05:15.329 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:05:15.329 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:05:15.329 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_5 2024-09-14T06:05:15.329 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:05:15.329 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:05:15.329 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_5 2024-09-14T06:05:15.329 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-14T06:05:15.329 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:05:15.373 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id=127e1aea6d17 2024-09-14T06:05:15.373 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:05:15.373 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:05:15.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_5 present 127e1aea6d17 2024-09-14T06:05:15.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:05:15.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:05:15.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-14T06:05:15.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=present 2024-09-14T06:05:15.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id=127e1aea6d17 2024-09-14T06:05:15.475 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:05:15.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:05:15.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_5 2024-09-14T06:05:15.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:05:15.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:05:15.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_5 2024-09-14T06:05:15.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-14T06:05:15.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:05:15.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id=127e1aea6d17 2024-09-14T06:05:15.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 127e1aea6d17 2024-09-14T06:05:15.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -z 127e1aea6d17 -o 127e1aea6d17 = 127e1aea6d17 2024-09-14T06:05:15.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=present 2024-09-14T06:05:15.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test present = present 2024-09-14T06:05:15.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:05:15.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ disable_mirror cluster2 mirror image_5 2024-09-14T06:05:15.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:05:15.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:05:15.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-14T06:05:15.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_5 2024-09-14T06:05:17.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:05:17.727+0000 7f1bc52a6640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55710d3031e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T06:05:17.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:05:17.727+0000 7f1bcb2b2640 -1 rbd::mirror::ImageReplayer: 0x55710cdb8a00 [3/ce7dc856-80fa-4c8d-a257-16e1dd6680e8] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T06:05:17.839 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-09-14T06:05:17.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-14T06:05:17.839 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-14T06:05:17.842 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-14T06:05:17.843 INFO:tasks.rbd_mirror_thrash:waiting for 76 secs before thrashing 2024-09-14T06:05:46.311 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-14T06:05:46.309+0000 7f9b14f28640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55f03a866000 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=189 2024-09-14T06:05:46.311 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-14T06:05:46.309+0000 7f9b14f28640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55f03a866000 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=193, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=9d38de88-cffd-4132-b7e8-809c69f2745a, primary_snap_id=bd, last_copied_object_number=128, snap_seqs={189=18446744073709551614}] 2024-09-14T06:05:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring disabled 2024-09-14T06:05:46.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2024-09-14T06:05:46.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:05:46.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:05:46.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-14T06:05:46.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:05:46.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:05:46.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:05:46.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:05:46.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_5 2024-09-14T06:05:46.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:05:46.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:05:46.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_5 2024-09-14T06:05:46.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-14T06:05:46.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:05:46.388 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-14T06:05:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:05:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:05:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:05:46.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2024-09-14T06:05:46.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:05:46.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:05:46.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-14T06:05:46.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:05:46.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:05:46.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:05:46.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:05:46.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_5 2024-09-14T06:05:46.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:05:46.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:05:46.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_5 2024-09-14T06:05:46.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-14T06:05:46.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:05:46.529 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-14T06:05:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:05:46.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:05:46.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:05:46.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:05:46.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ purge_snapshots cluster2 mirror image_5 2024-09-14T06:05:46.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:05:46.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:05:46.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-14T06:05:46.533 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap purge mirror/image_5 2024-09-14T06:05:46.578 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_5 2024-09-14T06:05:46.578 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:05:46.578 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:05:46.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-14T06:05:46.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:05:46.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:05:46.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_5 2024-09-14T06:05:46.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:05:46.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:05:46.579 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_5 2024-09-14T06:05:46.580 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-09-14T06:05:46.625 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-09-14T06:05:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-14T06:05:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:05:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-14T06:05:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_6 2024-09-14T06:05:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_6 512M 2024-09-14T06:05:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:05:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T06:05:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:05:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T06:05:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-14T06:05:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T06:05:46.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T06:05:46.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 512M ']' 2024-09-14T06:05:46.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=512M 2024-09-14T06:05:46.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T06:05:46.879 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_6 2024-09-14T06:05:46.930 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_6 2024-09-14T06:05:46.930 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:05:46.930 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:05:46.930 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-14T06:05:46.930 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T06:05:46.930 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2024-09-14T06:05:47.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T06:05:47.327 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2024-09-14T06:05:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_6': 2024-09-14T06:05:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 512 MiB in 128 objects 2024-09-14T06:05:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T06:05:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T06:05:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 19d973e58eaf 2024-09-14T06:05:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.19d973e58eaf 2024-09-14T06:05:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T06:05:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T06:05:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T06:05:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T06:05:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 06:05:46 2024 2024-09-14T06:05:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 06:05:46 2024 2024-09-14T06:05:47.368 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 06:05:46 2024 2024-09-14T06:05:47.368 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T06:05:47.368 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T06:05:47.368 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: d3be0992-7066-4e7a-b180-27adfc43a322 2024-09-14T06:05:47.368 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T06:05:47.370 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_6 2024-09-14T06:05:47.371 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:05:47.371 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:05:47.371 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-14T06:05:47.371 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:05:47.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=8 2024-09-14T06:05:47.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T06:05:47.372 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:05:47.627 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 45941760 bytes 2024-09-14T06:05:47.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 109303808 bytes 2024-09-14T06:05:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 126596096 bytes 2024-09-14T06:05:48.094 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 215593984 bytes 2024-09-14T06:05:48.225 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 266132480 bytes 2024-09-14T06:05:48.322 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 283413504 bytes 2024-09-14T06:05:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 297344000 bytes 2024-09-14T06:05:48.530 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 330560512 bytes 2024-09-14T06:05:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 375876608 bytes 2024-09-14T06:05:48.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 397570048 bytes 2024-09-14T06:05:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 398158848 bytes 2024-09-14T06:05:48.968 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 431820800 bytes 2024-09-14T06:05:49.083 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 479927296 bytes 2024-09-14T06:05:49.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 506042368 bytes 2024-09-14T06:05:49.285 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 516165632 bytes 2024-09-14T06:05:49.403 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 548948992 bytes 2024-09-14T06:05:49.518 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 588875776 bytes 2024-09-14T06:05:49.644 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 643937280 bytes 2024-09-14T06:05:49.762 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 688257024 bytes 2024-09-14T06:05:49.897 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 723995648 bytes 2024-09-14T06:05:49.977 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 724615168 bytes 2024-09-14T06:05:50.107 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 773506048 bytes 2024-09-14T06:05:50.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 826766336 bytes 2024-09-14T06:05:50.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 843058176 bytes 2024-09-14T06:05:50.459 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 891003904 bytes 2024-09-14T06:05:50.557 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 901060608 bytes 2024-09-14T06:05:50.672 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 934376448 bytes 2024-09-14T06:05:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 937358336 bytes 2024-09-14T06:05:50.926 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1021413376 bytes 2024-09-14T06:05:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1022016512 bytes 2024-09-14T06:05:51.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1049538560 bytes 2024-09-14T06:05:51.264 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1152891904 bytes 2024-09-14T06:05:51.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1176437760 bytes 2024-09-14T06:05:51.531 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1207410688 bytes 2024-09-14T06:05:51.681 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1279926272 bytes 2024-09-14T06:05:51.795 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1311993856 bytes 2024-09-14T06:05:51.898 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1329508352 bytes 2024-09-14T06:05:52.042 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1388628992 bytes 2024-09-14T06:05:52.143 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1404281856 bytes 2024-09-14T06:05:52.242 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1421003776 bytes 2024-09-14T06:05:52.369 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1465406464 bytes 2024-09-14T06:05:52.467 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1477002240 bytes 2024-09-14T06:05:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1495001088 bytes 2024-09-14T06:05:52.728 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1580721152 bytes 2024-09-14T06:05:52.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1625455616 bytes 2024-09-14T06:05:53.012 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1682777088 bytes 2024-09-14T06:05:53.138 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 1722778624 bytes 2024-09-14T06:05:53.294 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 1773894656 bytes 2024-09-14T06:05:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 1814792192 bytes 2024-09-14T06:05:53.527 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 1843854336 bytes 2024-09-14T06:05:53.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 1943073792 bytes 2024-09-14T06:05:53.811 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 1967790080 bytes 2024-09-14T06:05:53.918 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 1990543360 bytes 2024-09-14T06:05:54.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2011836416 bytes 2024-09-14T06:05:54.132 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2045682688 bytes 2024-09-14T06:05:54.247 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2102348800 bytes 2024-09-14T06:05:54.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2170569728 bytes 2024-09-14T06:05:54.516 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2190492672 bytes 2024-09-14T06:05:54.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2294949888 bytes 2024-09-14T06:05:54.820 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2340452352 bytes 2024-09-14T06:05:54.961 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2393801728 bytes 2024-09-14T06:05:55.096 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2437574656 bytes 2024-09-14T06:05:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2483516416 bytes 2024-09-14T06:05:55.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2492521472 bytes 2024-09-14T06:05:55.381 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T06:05:55.381 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T06:05:55.382 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:05:55.382 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:05:55.382 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_6 present 2024-09-14T06:05:55.382 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:05:55.382 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:05:55.382 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-14T06:05:55.382 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=present 2024-09-14T06:05:55.382 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:05:55.382 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:05:55.382 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:05:55.382 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_6 2024-09-14T06:05:55.382 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:05:55.382 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:05:55.383 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_6 2024-09-14T06:05:55.383 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-14T06:05:55.384 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:05:55.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id=127e150325e6 2024-09-14T06:05:55.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:05:55.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:05:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_6 present 127e150325e6 2024-09-14T06:05:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:05:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:05:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-14T06:05:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=present 2024-09-14T06:05:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id=127e150325e6 2024-09-14T06:05:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:05:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:05:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_6 2024-09-14T06:05:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:05:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:05:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_6 2024-09-14T06:05:55.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-14T06:05:55.537 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:05:55.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id=127e150325e6 2024-09-14T06:05:55.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 127e150325e6 2024-09-14T06:05:55.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -z 127e150325e6 -o 127e150325e6 = 127e150325e6 2024-09-14T06:05:55.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=present 2024-09-14T06:05:55.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test present = present 2024-09-14T06:05:55.582 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:05:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ disable_mirror cluster2 mirror image_6 2024-09-14T06:05:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:05:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:05:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-14T06:05:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_6 2024-09-14T06:06:17.643 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.775709812816352) > weight (0.25) 2024-09-14T06:06:17.644 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.9829395165311442) > weight (0.25) 2024-09-14T06:06:17.644 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-09-14T06:06:17.644 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-09-14T06:06:17.645 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-09-14T06:06:17.646 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-14T06:06:17.643+0000 7f4fab2bf640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 63162) UID: 1000 2024-09-14T06:06:17.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:06:17.726+0000 7f1bc4aa5640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55710d303a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T06:06:17.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:06:17.726+0000 7f1bcb2b2640 -1 rbd::mirror::ImageReplayer: 0x55710cdb8a00 [3/d3be0992-7066-4e7a-b180-27adfc43a322] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T06:06:20.646 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-09-14T06:06:20.646 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T06:06:20.646 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-09-14T06:06:20.647 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-14T06:06:20.647 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-14T06:06:20.650 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-14T06:06:20.650 INFO:tasks.rbd_mirror_thrash:waiting for 78 secs before thrashing 2024-09-14T06:06:27.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-14T06:06:27.218+0000 7f9b14727640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55f03bb74800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=190 2024-09-14T06:06:27.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-14T06:06:27.218+0000 7f9b14727640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55f03bb74800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=194, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=9d38de88-cffd-4132-b7e8-809c69f2745a, primary_snap_id=be, last_copied_object_number=128, snap_seqs={190=18446744073709551614}] 2024-09-14T06:06:27.227 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring disabled 2024-09-14T06:06:27.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2024-09-14T06:06:27.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:06:27.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:06:27.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-14T06:06:27.260 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:06:27.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:06:27.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:06:27.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:06:27.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_6 2024-09-14T06:06:27.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:06:27.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:06:27.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_6 2024-09-14T06:06:27.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-14T06:06:27.261 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:06:27.299 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-09-14T06:06:27.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:06:27.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:06:27.304 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:06:27.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2024-09-14T06:06:27.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:06:27.404 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:06:27.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-14T06:06:27.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:06:27.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:06:27.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:06:27.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:06:27.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_6 2024-09-14T06:06:27.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:06:27.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:06:27.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_6 2024-09-14T06:06:27.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-14T06:06:27.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:06:27.440 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-09-14T06:06:27.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:06:27.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:06:27.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:06:27.444 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:06:27.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ purge_snapshots cluster2 mirror image_6 2024-09-14T06:06:27.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:06:27.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:06:27.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-14T06:06:27.445 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap purge mirror/image_6 2024-09-14T06:06:27.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_6 2024-09-14T06:06:27.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:06:27.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:06:27.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-14T06:06:27.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:06:27.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:06:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_6 2024-09-14T06:06:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:06:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:06:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_6 2024-09-14T06:06:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-09-14T06:06:27.539 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-09-14T06:06:27.787 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-14T06:06:27.791 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:06:27.791 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-14T06:06:27.791 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_7 2024-09-14T06:06:27.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_7 512M 2024-09-14T06:06:27.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:06:27.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T06:06:27.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:06:27.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T06:06:27.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-14T06:06:27.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T06:06:27.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T06:06:27.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 512M ']' 2024-09-14T06:06:27.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=512M 2024-09-14T06:06:27.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T06:06:27.792 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_7 2024-09-14T06:06:27.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_7 2024-09-14T06:06:27.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:06:27.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:06:27.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-14T06:06:27.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T06:06:27.845 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2024-09-14T06:06:28.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T06:06:28.248 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2024-09-14T06:06:28.288 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_7': 2024-09-14T06:06:28.288 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 512 MiB in 128 objects 2024-09-14T06:06:28.288 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T06:06:28.288 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T06:06:28.288 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 19fb55888921 2024-09-14T06:06:28.288 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.19fb55888921 2024-09-14T06:06:28.288 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T06:06:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T06:06:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T06:06:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T06:06:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 06:06:27 2024 2024-09-14T06:06:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 06:06:27 2024 2024-09-14T06:06:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 06:06:27 2024 2024-09-14T06:06:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T06:06:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T06:06:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: e5f2284c-adfd-45dd-8197-50148ee30829 2024-09-14T06:06:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T06:06:28.291 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_7 2024-09-14T06:06:28.292 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:06:28.292 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:06:28.292 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-14T06:06:28.292 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:06:28.293 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=14 2024-09-14T06:06:28.294 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T06:06:28.294 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:06:28.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 9311232 bytes 2024-09-14T06:06:28.668 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 41162752 bytes 2024-09-14T06:06:28.783 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 63337472 bytes 2024-09-14T06:06:28.910 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 108032000 bytes 2024-09-14T06:06:29.093 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 212566016 bytes 2024-09-14T06:06:29.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 294681600 bytes 2024-09-14T06:06:29.385 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 339187712 bytes 2024-09-14T06:06:29.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 395068416 bytes 2024-09-14T06:06:29.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 444444672 bytes 2024-09-14T06:06:29.801 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 523918336 bytes 2024-09-14T06:06:29.952 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 595184640 bytes 2024-09-14T06:06:30.063 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 624781312 bytes 2024-09-14T06:06:30.173 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 653044736 bytes 2024-09-14T06:06:30.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 707265536 bytes 2024-09-14T06:06:30.401 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 719926272 bytes 2024-09-14T06:06:30.559 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 805628928 bytes 2024-09-14T06:06:30.707 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 864468992 bytes 2024-09-14T06:06:30.887 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 967545856 bytes 2024-09-14T06:06:31.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 1062177792 bytes 2024-09-14T06:06:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 1149742080 bytes 2024-09-14T06:06:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 1174784000 bytes 2024-09-14T06:06:31.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 1247137792 bytes 2024-09-14T06:06:31.588 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1248925696 bytes 2024-09-14T06:06:31.713 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1296876544 bytes 2024-09-14T06:06:31.813 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1315428352 bytes 2024-09-14T06:06:31.950 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1373205504 bytes 2024-09-14T06:06:32.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1428989952 bytes 2024-09-14T06:06:32.270 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1526352896 bytes 2024-09-14T06:06:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1534557184 bytes 2024-09-14T06:06:32.484 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1574832128 bytes 2024-09-14T06:06:32.610 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1626283008 bytes 2024-09-14T06:06:32.784 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1717197824 bytes 2024-09-14T06:06:32.876 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1731165184 bytes 2024-09-14T06:06:33.029 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1800791040 bytes 2024-09-14T06:06:33.117 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1801407488 bytes 2024-09-14T06:06:33.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1825873920 bytes 2024-09-14T06:06:33.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1889359872 bytes 2024-09-14T06:06:33.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1942505472 bytes 2024-09-14T06:06:33.638 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1989941248 bytes 2024-09-14T06:06:33.746 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 2017894400 bytes 2024-09-14T06:06:33.843 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-14T06:06:33.843 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-14T06:06:33.844 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-09-14T06:06:33.845 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-14T06:06:33.843+0000 7fde8e780640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 69138) UID: 1000 2024-09-14T06:06:33.854 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 2067615744 bytes 2024-09-14T06:06:34.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 2204065792 bytes 2024-09-14T06:06:34.286 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 2308917248 bytes 2024-09-14T06:06:34.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 2356699136 bytes 2024-09-14T06:06:34.588 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 2445840384 bytes 2024-09-14T06:06:34.688 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 2472049664 bytes 2024-09-14T06:06:34.811 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 2511580160 bytes 2024-09-14T06:06:34.917 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2538194944 bytes 2024-09-14T06:06:35.021 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2563161088 bytes 2024-09-14T06:06:35.246 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2709842944 bytes 2024-09-14T06:06:35.332 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2710483968 bytes 2024-09-14T06:06:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2759653376 bytes 2024-09-14T06:06:35.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2774070272 bytes 2024-09-14T06:06:35.698 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2839699456 bytes 2024-09-14T06:06:35.849 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2918721536 bytes 2024-09-14T06:06:36.004 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 3012501504 bytes 2024-09-14T06:06:36.140 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 3035737088 bytes 2024-09-14T06:06:36.263 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 3108307968 bytes 2024-09-14T06:06:36.421 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 3151832064 bytes 2024-09-14T06:06:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 3216431104 bytes 2024-09-14T06:06:36.669 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 3247819776 bytes 2024-09-14T06:06:36.765 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 3248421888 bytes 2024-09-14T06:06:36.873 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 3275377664 bytes 2024-09-14T06:06:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 3329360896 bytes 2024-09-14T06:06:37.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 3356690432 bytes 2024-09-14T06:06:37.250 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 3404136448 bytes 2024-09-14T06:06:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 3465702400 bytes 2024-09-14T06:06:37.485 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 3480103936 bytes 2024-09-14T06:06:37.598 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 3512690688 bytes 2024-09-14T06:06:37.742 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 3563325440 bytes 2024-09-14T06:06:37.911 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 3650675712 bytes 2024-09-14T06:06:38.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 3673289728 bytes 2024-09-14T06:06:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7300 IOs, 3726842880 bytes 2024-09-14T06:06:38.268 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7400 IOs, 3754115072 bytes 2024-09-14T06:06:38.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7500 IOs, 3820153856 bytes 2024-09-14T06:06:38.551 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7600 IOs, 3885540352 bytes 2024-09-14T06:06:38.655 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7700 IOs, 3892159488 bytes 2024-09-14T06:06:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7800 IOs, 3931762688 bytes 2024-09-14T06:06:38.899 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7900 IOs, 3982716928 bytes 2024-09-14T06:06:38.995 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8000 IOs, 3998745600 bytes 2024-09-14T06:06:39.104 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8100 IOs, 4025120768 bytes 2024-09-14T06:06:39.251 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8200 IOs, 4100168704 bytes 2024-09-14T06:06:39.373 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8300 IOs, 4130989056 bytes 2024-09-14T06:06:39.488 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8400 IOs, 4164464640 bytes 2024-09-14T06:06:39.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8500 IOs, 4217893888 bytes 2024-09-14T06:06:39.721 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8600 IOs, 4237251584 bytes 2024-09-14T06:06:39.839 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8700 IOs, 4278998016 bytes 2024-09-14T06:06:39.844 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-14T06:06:39.844 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T06:06:39.844 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-14T06:06:39.844 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-14T06:06:39.845 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-14T06:06:39.847 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-14T06:06:39.848 INFO:tasks.rbd_mirror_thrash:waiting for 100 secs before thrashing 2024-09-14T06:06:40.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8800 IOs, 4360161280 bytes 2024-09-14T06:06:40.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:8900 IOs, 4390028288 bytes 2024-09-14T06:06:40.251 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9000 IOs, 4401586176 bytes 2024-09-14T06:06:40.382 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9100 IOs, 4463860736 bytes 2024-09-14T06:06:40.504 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9200 IOs, 4484295680 bytes 2024-09-14T06:06:40.611 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9300 IOs, 4514767872 bytes 2024-09-14T06:06:40.731 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9400 IOs, 4568672256 bytes 2024-09-14T06:06:40.914 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9500 IOs, 4647843840 bytes 2024-09-14T06:06:41.057 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9600 IOs, 4709438464 bytes 2024-09-14T06:06:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9700 IOs, 4747924480 bytes 2024-09-14T06:06:41.272 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9800 IOs, 4757039104 bytes 2024-09-14T06:06:41.382 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:9900 IOs, 4787453952 bytes 2024-09-14T06:06:41.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10000 IOs, 4816689152 bytes 2024-09-14T06:06:41.613 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10100 IOs, 4854803456 bytes 2024-09-14T06:06:41.711 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10200 IOs, 4868642816 bytes 2024-09-14T06:06:41.826 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10300 IOs, 4898383872 bytes 2024-09-14T06:06:41.944 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10400 IOs, 4932052992 bytes 2024-09-14T06:06:42.125 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10500 IOs, 5027351552 bytes 2024-09-14T06:06:42.249 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:10600 IOs, 5068132352 bytes 2024-09-14T06:06:42.299 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T06:06:42.299 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T06:06:42.299 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:06:42.299 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:06:42.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_7 present 2024-09-14T06:06:42.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:06:42.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:06:42.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-14T06:06:42.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=present 2024-09-14T06:06:42.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:06:42.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:06:42.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:06:42.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_7 2024-09-14T06:06:42.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:06:42.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:06:42.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_7 2024-09-14T06:06:42.300 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-14T06:06:42.301 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:06:42.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id=127e2e735de0 2024-09-14T06:06:42.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:06:42.346 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:06:42.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_7 present 127e2e735de0 2024-09-14T06:06:42.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:06:42.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:06:42.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-14T06:06:42.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=present 2024-09-14T06:06:42.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id=127e2e735de0 2024-09-14T06:06:42.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:06:42.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:06:42.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_7 2024-09-14T06:06:42.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:06:42.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:06:42.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_7 2024-09-14T06:06:42.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-14T06:06:42.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:06:42.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id=127e2e735de0 2024-09-14T06:06:42.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 127e2e735de0 2024-09-14T06:06:42.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -z 127e2e735de0 -o 127e2e735de0 = 127e2e735de0 2024-09-14T06:06:42.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=present 2024-09-14T06:06:42.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test present = present 2024-09-14T06:06:42.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:06:42.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ disable_mirror cluster2 mirror image_7 2024-09-14T06:06:42.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:06:42.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:06:42.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-14T06:06:42.494 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_7 2024-09-14T06:06:47.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:06:47.728+0000 7f1bc52a6640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55710c4f6000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T06:06:47.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:06:47.728+0000 7f1bcb2b2640 -1 rbd::mirror::ImageReplayer: 0x55710cdb9b80 [3/e5f2284c-adfd-45dd-8197-50148ee30829] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T06:07:13.288 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring disabled 2024-09-14T06:07:13.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2024-09-14T06:07:13.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:07:13.309 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:07:13.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-14T06:07:13.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:07:13.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:07:13.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:07:13.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:07:13.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_7 2024-09-14T06:07:13.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:07:13.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:07:13.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_7 2024-09-14T06:07:13.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-14T06:07:13.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:07:13.350 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-14T06:07:13.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:07:13.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:07:13.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:07:13.456 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2024-09-14T06:07:13.456 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:07:13.456 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:07:13.456 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-14T06:07:13.456 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:07:13.456 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:07:13.456 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:07:13.457 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:07:13.457 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_7 2024-09-14T06:07:13.457 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:07:13.457 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:07:13.457 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_7 2024-09-14T06:07:13.457 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-14T06:07:13.457 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:07:13.492 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-14T06:07:13.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:07:13.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:07:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:07:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:07:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ purge_snapshots cluster2 mirror image_7 2024-09-14T06:07:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:07:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:07:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-14T06:07:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap purge mirror/image_7 2024-09-14T06:07:13.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_7 2024-09-14T06:07:13.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:07:13.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:07:13.542 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-14T06:07:13.542 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:07:13.542 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:07:13.542 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_7 2024-09-14T06:07:13.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:07:13.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:07:13.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_7 2024-09-14T06:07:13.543 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-09-14T06:07:13.589 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-09-14T06:07:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-14T06:07:13.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:07:13.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-14T06:07:13.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_8 2024-09-14T06:07:13.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_8 512M 2024-09-14T06:07:13.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:07:13.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T06:07:13.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:07:13.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T06:07:13.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-14T06:07:13.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T06:07:13.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T06:07:13.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 512M ']' 2024-09-14T06:07:13.843 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=512M 2024-09-14T06:07:13.844 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T06:07:13.844 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_8 2024-09-14T06:07:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_8 2024-09-14T06:07:13.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:07:13.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:07:13.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-14T06:07:13.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T06:07:13.896 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2024-09-14T06:07:14.299 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T06:07:14.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2024-09-14T06:07:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_8': 2024-09-14T06:07:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 512 MiB in 128 objects 2024-09-14T06:07:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T06:07:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T06:07:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 1a1944a6ca80 2024-09-14T06:07:14.350 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.1a1944a6ca80 2024-09-14T06:07:14.350 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T06:07:14.350 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T06:07:14.350 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T06:07:14.350 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T06:07:14.350 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 06:07:13 2024 2024-09-14T06:07:14.350 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 06:07:13 2024 2024-09-14T06:07:14.350 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 06:07:13 2024 2024-09-14T06:07:14.350 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T06:07:14.350 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T06:07:14.350 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: f1d42f48-ccc8-412d-8f02-6d13166635bf 2024-09-14T06:07:14.350 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T06:07:14.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_8 2024-09-14T06:07:14.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:07:14.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:07:14.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-14T06:07:14.353 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:07:14.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=9 2024-09-14T06:07:14.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T06:07:14.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:07:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 53905408 bytes 2024-09-14T06:07:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 96173056 bytes 2024-09-14T06:07:14.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 178828288 bytes 2024-09-14T06:07:15.041 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 195263488 bytes 2024-09-14T06:07:15.127 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 195865600 bytes 2024-09-14T06:07:15.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 208472064 bytes 2024-09-14T06:07:15.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 242428928 bytes 2024-09-14T06:07:15.473 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 296809472 bytes 2024-09-14T06:07:15.641 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 398862336 bytes 2024-09-14T06:07:15.777 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 444736512 bytes 2024-09-14T06:07:15.870 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 456079360 bytes 2024-09-14T06:07:15.970 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 472343552 bytes 2024-09-14T06:07:16.103 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 528506880 bytes 2024-09-14T06:07:16.191 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 562494464 bytes 2024-09-14T06:07:16.322 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 596381696 bytes 2024-09-14T06:07:16.465 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 660766720 bytes 2024-09-14T06:07:16.562 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 672780288 bytes 2024-09-14T06:07:16.673 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 703745024 bytes 2024-09-14T06:07:16.771 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 731436032 bytes 2024-09-14T06:07:16.909 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 781541376 bytes 2024-09-14T06:07:17.044 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 840862720 bytes 2024-09-14T06:07:17.155 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 876111872 bytes 2024-09-14T06:07:17.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 954165248 bytes 2024-09-14T06:07:17.459 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1021864960 bytes 2024-09-14T06:07:17.650 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1126041600 bytes 2024-09-14T06:07:17.786 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1184454656 bytes 2024-09-14T06:07:17.893 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1216929792 bytes 2024-09-14T06:07:18.036 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1276895232 bytes 2024-09-14T06:07:18.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1373810688 bytes 2024-09-14T06:07:18.324 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1406268416 bytes 2024-09-14T06:07:18.438 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1443364864 bytes 2024-09-14T06:07:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1465402368 bytes 2024-09-14T06:07:18.662 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1503111168 bytes 2024-09-14T06:07:18.779 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1539651584 bytes 2024-09-14T06:07:18.931 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1612220416 bytes 2024-09-14T06:07:19.035 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1637656576 bytes 2024-09-14T06:07:19.156 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1681401856 bytes 2024-09-14T06:07:19.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1689860096 bytes 2024-09-14T06:07:19.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1714044928 bytes 2024-09-14T06:07:19.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1747119104 bytes 2024-09-14T06:07:19.603 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1800502272 bytes 2024-09-14T06:07:19.716 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1825216512 bytes 2024-09-14T06:07:19.885 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1922828288 bytes 2024-09-14T06:07:20.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1982267392 bytes 2024-09-14T06:07:20.177 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 2057113600 bytes 2024-09-14T06:07:20.293 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 2090247168 bytes 2024-09-14T06:07:20.434 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 2153209856 bytes 2024-09-14T06:07:20.538 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2171510784 bytes 2024-09-14T06:07:20.666 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2214113280 bytes 2024-09-14T06:07:20.804 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2267997184 bytes 2024-09-14T06:07:20.938 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2314769408 bytes 2024-09-14T06:07:21.077 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2378632192 bytes 2024-09-14T06:07:21.210 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2433948672 bytes 2024-09-14T06:07:21.363 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2522037248 bytes 2024-09-14T06:07:21.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2581662720 bytes 2024-09-14T06:07:21.637 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2611078144 bytes 2024-09-14T06:07:21.757 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2652854272 bytes 2024-09-14T06:07:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2700544000 bytes 2024-09-14T06:07:22.131 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2749208576 bytes 2024-09-14T06:07:22.239 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2789206016 bytes 2024-09-14T06:07:22.417 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2872711168 bytes 2024-09-14T06:07:22.520 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2892768256 bytes 2024-09-14T06:07:22.692 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2931700736 bytes 2024-09-14T06:07:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2967657472 bytes 2024-09-14T06:07:22.953 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2971996160 bytes 2024-09-14T06:07:23.071 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 3015471104 bytes 2024-09-14T06:07:23.223 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 3083098112 bytes 2024-09-14T06:07:23.361 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T06:07:23.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T06:07:23.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:07:23.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:07:23.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_8 present 2024-09-14T06:07:23.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:07:23.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:07:23.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-14T06:07:23.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=present 2024-09-14T06:07:23.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:07:23.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:07:23.362 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:07:23.363 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_8 2024-09-14T06:07:23.363 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:07:23.363 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:07:23.363 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_8 2024-09-14T06:07:23.363 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-14T06:07:23.363 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:07:23.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id=127e330ade9b 2024-09-14T06:07:23.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:07:23.409 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:07:23.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_8 present 127e330ade9b 2024-09-14T06:07:23.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:07:23.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:07:23.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-14T06:07:23.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=present 2024-09-14T06:07:23.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id=127e330ade9b 2024-09-14T06:07:23.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:07:23.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:07:23.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_8 2024-09-14T06:07:23.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:07:23.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:07:23.513 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_8 2024-09-14T06:07:23.514 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-14T06:07:23.514 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:07:23.556 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id=127e330ade9b 2024-09-14T06:07:23.557 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 127e330ade9b 2024-09-14T06:07:23.557 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -z 127e330ade9b -o 127e330ade9b = 127e330ade9b 2024-09-14T06:07:23.557 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=present 2024-09-14T06:07:23.557 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test present = present 2024-09-14T06:07:23.557 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:07:23.557 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ disable_mirror cluster2 mirror image_8 2024-09-14T06:07:23.557 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:07:23.557 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:07:23.557 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-14T06:07:23.557 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_8 2024-09-14T06:07:38.651 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.3833353375774676) > weight (0.25) 2024-09-14T06:07:38.652 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-09-14T06:07:38.652 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-09-14T06:07:38.652 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-09-14T06:07:38.653 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:07:38.652+0000 7f1bcc2b4640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 45431) UID: 1000 2024-09-14T06:07:42.712 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-14T06:07:42.711+0000 7fc6b05c7640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558ca7f97040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T06:07:42.712 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-14T06:07:42.711+0000 7fc6b6dd4640 -1 rbd::mirror::ImageReplayer: 0x558ca80e8280 [3/f1d42f48-ccc8-412d-8f02-6d13166635bf] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T06:07:43.653 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-09-14T06:07:43.653 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T06:07:43.653 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-09-14T06:07:43.653 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-14T06:07:43.654 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-14T06:07:43.657 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-14T06:07:43.657 INFO:tasks.rbd_mirror_thrash:waiting for 84 secs before thrashing 2024-09-14T06:07:53.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-14T06:07:53.981+0000 7f9b14727640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55f039b4d000 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=192 2024-09-14T06:07:53.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-14T06:07:53.981+0000 7f9b14727640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55f039b4d000 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=196, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=9d38de88-cffd-4132-b7e8-809c69f2745a, primary_snap_id=c0, last_copied_object_number=128, snap_seqs={192=18446744073709551614}] 2024-09-14T06:07:54.002 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring disabled 2024-09-14T06:07:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2024-09-14T06:07:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:07:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:07:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-14T06:07:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:07:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:07:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:07:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:07:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_8 2024-09-14T06:07:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:07:54.021 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:07:54.021 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_8 2024-09-14T06:07:54.021 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-14T06:07:54.021 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:07:54.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-14T06:07:54.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:07:54.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:07:54.065 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:07:54.166 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2024-09-14T06:07:54.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:07:54.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:07:54.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-14T06:07:54.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:07:54.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:07:54.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:07:54.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:07:54.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_8 2024-09-14T06:07:54.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:07:54.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:07:54.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_8 2024-09-14T06:07:54.167 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-14T06:07:54.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:07:54.202 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-14T06:07:54.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:07:54.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:07:54.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:07:54.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:07:54.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ purge_snapshots cluster2 mirror image_8 2024-09-14T06:07:54.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:07:54.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:07:54.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-14T06:07:54.206 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap purge mirror/image_8 2024-09-14T06:07:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_8 2024-09-14T06:07:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:07:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:07:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-14T06:07:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:07:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:07:54.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_8 2024-09-14T06:07:54.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:07:54.253 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:07:54.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_8 2024-09-14T06:07:54.254 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2024-09-14T06:07:54.299 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2024-09-14T06:07:54.548 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-14T06:07:54.552 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:07:54.552 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-14T06:07:54.552 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_9 2024-09-14T06:07:54.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_9 512M 2024-09-14T06:07:54.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:07:54.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T06:07:54.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:07:54.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T06:07:54.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-14T06:07:54.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T06:07:54.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T06:07:54.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 512M ']' 2024-09-14T06:07:54.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=512M 2024-09-14T06:07:54.553 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T06:07:54.554 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_9 2024-09-14T06:07:54.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_9 2024-09-14T06:07:54.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:07:54.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:07:54.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-14T06:07:54.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T06:07:54.605 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2024-09-14T06:07:54.998 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T06:07:55.008 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2024-09-14T06:07:55.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_9': 2024-09-14T06:07:55.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 512 MiB in 128 objects 2024-09-14T06:07:55.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T06:07:55.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T06:07:55.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 1a3a5a5540b1 2024-09-14T06:07:55.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.1a3a5a5540b1 2024-09-14T06:07:55.048 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T06:07:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T06:07:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T06:07:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T06:07:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 06:07:54 2024 2024-09-14T06:07:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 06:07:54 2024 2024-09-14T06:07:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 06:07:54 2024 2024-09-14T06:07:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T06:07:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T06:07:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: d01f79ba-ba56-4bd8-897a-4c8a3ef8a725 2024-09-14T06:07:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T06:07:55.051 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_9 2024-09-14T06:07:55.051 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:07:55.051 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:07:55.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-14T06:07:55.052 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:07:55.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=9 2024-09-14T06:07:55.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T06:07:55.053 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:07:55.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 647168 bytes 2024-09-14T06:07:55.405 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 34924544 bytes 2024-09-14T06:07:55.562 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 92322816 bytes 2024-09-14T06:07:55.703 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 158240768 bytes 2024-09-14T06:07:55.833 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 217483264 bytes 2024-09-14T06:07:55.991 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 282137600 bytes 2024-09-14T06:07:56.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 315144192 bytes 2024-09-14T06:07:56.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 346921984 bytes 2024-09-14T06:07:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 404452352 bytes 2024-09-14T06:07:56.499 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 472265728 bytes 2024-09-14T06:07:56.625 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 521693184 bytes 2024-09-14T06:07:56.732 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 555668480 bytes 2024-09-14T06:07:56.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 579713024 bytes 2024-09-14T06:07:56.941 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 600832000 bytes 2024-09-14T06:07:57.064 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 668471296 bytes 2024-09-14T06:07:57.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 727470080 bytes 2024-09-14T06:07:57.341 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 770455552 bytes 2024-09-14T06:07:57.476 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 826350592 bytes 2024-09-14T06:07:57.561 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 826951680 bytes 2024-09-14T06:07:57.703 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 887366656 bytes 2024-09-14T06:07:57.806 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 907846656 bytes 2024-09-14T06:07:57.949 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 973490176 bytes 2024-09-14T06:07:58.039 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 980744192 bytes 2024-09-14T06:07:58.188 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1046261760 bytes 2024-09-14T06:07:58.287 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1056886784 bytes 2024-09-14T06:07:58.395 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1080212480 bytes 2024-09-14T06:07:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1102214144 bytes 2024-09-14T06:07:58.589 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1112556544 bytes 2024-09-14T06:07:58.678 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1116142592 bytes 2024-09-14T06:07:58.810 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1164623872 bytes 2024-09-14T06:07:58.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1181921280 bytes 2024-09-14T06:07:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1202927616 bytes 2024-09-14T06:07:59.195 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1301624832 bytes 2024-09-14T06:07:59.329 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1362707456 bytes 2024-09-14T06:07:59.473 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1431474176 bytes 2024-09-14T06:07:59.587 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1463150592 bytes 2024-09-14T06:07:59.727 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1540020224 bytes 2024-09-14T06:07:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1553867776 bytes 2024-09-14T06:07:59.933 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1564637184 bytes 2024-09-14T06:08:00.090 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1652151296 bytes 2024-09-14T06:08:00.219 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1698639872 bytes 2024-09-14T06:08:00.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1721194496 bytes 2024-09-14T06:08:00.400 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1725908992 bytes 2024-09-14T06:08:00.562 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1802418176 bytes 2024-09-14T06:08:00.687 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1851761664 bytes 2024-09-14T06:08:00.819 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 1899825152 bytes 2024-09-14T06:08:00.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 1968547840 bytes 2024-09-14T06:08:01.082 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2001989632 bytes 2024-09-14T06:08:01.176 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2009543680 bytes 2024-09-14T06:08:01.314 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2074843136 bytes 2024-09-14T06:08:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2091115520 bytes 2024-09-14T06:08:01.524 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2123114496 bytes 2024-09-14T06:08:01.642 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2160106496 bytes 2024-09-14T06:08:01.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2199105536 bytes 2024-09-14T06:08:01.912 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2274673664 bytes 2024-09-14T06:08:02.051 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5600 IOs, 2330712064 bytes 2024-09-14T06:08:02.136 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5700 IOs, 2331331584 bytes 2024-09-14T06:08:02.250 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5800 IOs, 2368069632 bytes 2024-09-14T06:08:02.344 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5900 IOs, 2368696320 bytes 2024-09-14T06:08:02.447 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6000 IOs, 2390701056 bytes 2024-09-14T06:08:02.572 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6100 IOs, 2431476736 bytes 2024-09-14T06:08:02.776 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6200 IOs, 2572077056 bytes 2024-09-14T06:08:02.884 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6300 IOs, 2597624832 bytes 2024-09-14T06:08:03.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6400 IOs, 2667902976 bytes 2024-09-14T06:08:03.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6500 IOs, 2711692288 bytes 2024-09-14T06:08:03.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6600 IOs, 2747260928 bytes 2024-09-14T06:08:03.365 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6700 IOs, 2747900928 bytes 2024-09-14T06:08:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6800 IOs, 2768102400 bytes 2024-09-14T06:08:03.584 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:6900 IOs, 2808028160 bytes 2024-09-14T06:08:03.673 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7000 IOs, 2808633344 bytes 2024-09-14T06:08:03.775 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7100 IOs, 2827281408 bytes 2024-09-14T06:08:03.974 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:7200 IOs, 2947971072 bytes 2024-09-14T06:08:04.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T06:08:04.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T06:08:04.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:08:04.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:08:04.059 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_9 present 2024-09-14T06:08:04.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:08:04.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:08:04.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-14T06:08:04.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=present 2024-09-14T06:08:04.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:08:04.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:08:04.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:08:04.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_9 2024-09-14T06:08:04.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:08:04.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:08:04.060 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_9 2024-09-14T06:08:04.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-14T06:08:04.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:08:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id=127ea7fed9a2 2024-09-14T06:08:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:08:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:08:04.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_9 present 127ea7fed9a2 2024-09-14T06:08:04.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:08:04.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:08:04.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-14T06:08:04.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=present 2024-09-14T06:08:04.212 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id=127ea7fed9a2 2024-09-14T06:08:04.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:08:04.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:08:04.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_9 2024-09-14T06:08:04.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:08:04.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:08:04.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_9 2024-09-14T06:08:04.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-14T06:08:04.213 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:08:04.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id=127ea7fed9a2 2024-09-14T06:08:04.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 127ea7fed9a2 2024-09-14T06:08:04.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -z 127ea7fed9a2 -o 127ea7fed9a2 = 127ea7fed9a2 2024-09-14T06:08:04.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=present 2024-09-14T06:08:04.258 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test present = present 2024-09-14T06:08:04.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:08:04.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ disable_mirror cluster2 mirror image_9 2024-09-14T06:08:04.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:08:04.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:08:04.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-14T06:08:04.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_9 2024-09-14T06:08:19.848 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8221805674438709) > weight (0.25) 2024-09-14T06:08:19.849 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.8263270705654113) > weight (0.25) 2024-09-14T06:08:19.849 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-09-14T06:08:19.849 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-09-14T06:08:19.849 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2024-09-14T06:08:19.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-14T06:08:19.849+0000 7fbe425f4640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 53937) UID: 1000 2024-09-14T06:08:21.851 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-09-14T06:08:21.851 INFO:teuthology.orchestra.run:waiting for 600 2024-09-14T06:08:21.852 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-09-14T06:08:21.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-09-14T06:08:21.852 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-09-14T06:08:21.856 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-09-14T06:08:21.856 INFO:tasks.rbd_mirror_thrash:waiting for 89 secs before thrashing 2024-09-14T06:08:30.771 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-14T06:08:30.769+0000 7fc6afdc6640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558ca7f971e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T06:08:30.771 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-14T06:08:30.769+0000 7fc6b65d3640 -1 rbd::mirror::ImageReplayer: 0x558ca80e8500 [3/d01f79ba-ba56-4bd8-897a-4c8a3ef8a725] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T06:08:35.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-14T06:08:35.240+0000 7f9b14f28640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55f03a10e000 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=193 2024-09-14T06:08:35.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-14T06:08:35.240+0000 7f9b14f28640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55f03a10e000 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=197, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=9d38de88-cffd-4132-b7e8-809c69f2745a, primary_snap_id=c1, last_copied_object_number=128, snap_seqs={193=18446744073709551614}] 2024-09-14T06:08:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring disabled 2024-09-14T06:08:35.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2024-09-14T06:08:35.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:08:35.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:08:35.282 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-14T06:08:35.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:08:35.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:08:35.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:08:35.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:08:35.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_9 2024-09-14T06:08:35.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:08:35.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:08:35.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_9 2024-09-14T06:08:35.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-14T06:08:35.283 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:08:35.322 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-14T06:08:35.325 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:08:35.326 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:08:35.326 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:08:35.427 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2024-09-14T06:08:35.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:08:35.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:08:35.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-14T06:08:35.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:08:35.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:08:35.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:08:35.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:08:35.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_9 2024-09-14T06:08:35.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:08:35.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:08:35.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_9 2024-09-14T06:08:35.428 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-14T06:08:35.429 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:08:35.464 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-14T06:08:35.468 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:08:35.468 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:08:35.468 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:08:35.468 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:08:35.468 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ purge_snapshots cluster2 mirror image_9 2024-09-14T06:08:35.469 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:08:35.469 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:08:35.469 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-14T06:08:35.469 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap purge mirror/image_9 2024-09-14T06:08:35.515 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_9 2024-09-14T06:08:35.515 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:08:35.515 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:08:35.515 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-14T06:08:35.515 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:08:35.515 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:08:35.516 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_9 2024-09-14T06:08:35.516 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:08:35.516 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:08:35.516 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_9 2024-09-14T06:08:35.516 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2024-09-14T06:08:35.562 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2024-09-14T06:08:35.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-14T06:08:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:08:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for i in `seq 1 10` 2024-09-14T06:08:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image=image_10 2024-09-14T06:08:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ create_image cluster2 mirror image_10 512M 2024-09-14T06:08:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:08:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T06:08:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:08:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T06:08:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-14T06:08:35.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T06:08:35.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local size=128 2024-09-14T06:08:35.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -n 512M ']' 2024-09-14T06:08:35.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ size=512M 2024-09-14T06:08:35.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ shift 2024-09-14T06:08:35.883 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image_10 2024-09-14T06:08:35.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ enable_mirror cluster2 mirror image_10 2024-09-14T06:08:35.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:08:35.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:08:35.934 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-14T06:08:35.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local mode=snapshot 2024-09-14T06:08:35.935 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2024-09-14T06:08:36.257 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring enabled 2024-09-14T06:08:36.267 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2024-09-14T06:08:36.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:rbd image 'image_10': 2024-09-14T06:08:36.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: size 512 MiB in 128 objects 2024-09-14T06:08:36.306 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: order 22 (4 MiB objects) 2024-09-14T06:08:36.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: snapshot_count: 1 2024-09-14T06:08:36.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: id: 1a583025d991 2024-09-14T06:08:36.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: block_name_prefix: rbd_data.1a583025d991 2024-09-14T06:08:36.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: format: 2 2024-09-14T06:08:36.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-14T06:08:36.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: op_features: 2024-09-14T06:08:36.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: flags: 2024-09-14T06:08:36.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: create_timestamp: Sat Sep 14 06:08:35 2024 2024-09-14T06:08:36.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: access_timestamp: Sat Sep 14 06:08:35 2024 2024-09-14T06:08:36.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: modify_timestamp: Sat Sep 14 06:08:35 2024 2024-09-14T06:08:36.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring state: enabled 2024-09-14T06:08:36.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring mode: snapshot 2024-09-14T06:08:36.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring global id: 7fc24725-16ee-4027-a514-0ffde98759ee 2024-09-14T06:08:36.307 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout: mirroring primary: true 2024-09-14T06:08:36.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stress_write_image cluster2 mirror image_10 2024-09-14T06:08:36.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:08:36.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:08:36.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-14T06:08:36.310 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-14T06:08:36.312 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local duration=7 2024-09-14T06:08:36.312 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T06:08:36.312 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2024-09-14T06:08:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:100 IOs, 52864000 bytes 2024-09-14T06:08:36.722 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:200 IOs, 88211456 bytes 2024-09-14T06:08:36.907 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:300 IOs, 191048704 bytes 2024-09-14T06:08:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:400 IOs, 198435840 bytes 2024-09-14T06:08:37.113 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:500 IOs, 229661696 bytes 2024-09-14T06:08:37.226 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:600 IOs, 281368576 bytes 2024-09-14T06:08:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:700 IOs, 305155072 bytes 2024-09-14T06:08:37.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:800 IOs, 341513216 bytes 2024-09-14T06:08:37.595 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:900 IOs, 390798336 bytes 2024-09-14T06:08:37.702 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1000 IOs, 430063616 bytes 2024-09-14T06:08:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1100 IOs, 479286272 bytes 2024-09-14T06:08:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1200 IOs, 560226304 bytes 2024-09-14T06:08:38.105 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1300 IOs, 577172480 bytes 2024-09-14T06:08:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1400 IOs, 648059904 bytes 2024-09-14T06:08:38.359 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1500 IOs, 672483328 bytes 2024-09-14T06:08:38.474 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1600 IOs, 709925888 bytes 2024-09-14T06:08:38.573 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1700 IOs, 728546304 bytes 2024-09-14T06:08:38.700 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1800 IOs, 795330560 bytes 2024-09-14T06:08:38.837 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:1900 IOs, 843066368 bytes 2024-09-14T06:08:38.976 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2000 IOs, 912012288 bytes 2024-09-14T06:08:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2100 IOs, 1016665088 bytes 2024-09-14T06:08:39.269 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2200 IOs, 1041775616 bytes 2024-09-14T06:08:39.378 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2300 IOs, 1071839232 bytes 2024-09-14T06:08:39.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2400 IOs, 1097373696 bytes 2024-09-14T06:08:39.620 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2500 IOs, 1154341888 bytes 2024-09-14T06:08:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2600 IOs, 1209477120 bytes 2024-09-14T06:08:39.878 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2700 IOs, 1268506624 bytes 2024-09-14T06:08:40.023 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2800 IOs, 1329181696 bytes 2024-09-14T06:08:40.148 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:2900 IOs, 1362786304 bytes 2024-09-14T06:08:40.256 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3000 IOs, 1392979968 bytes 2024-09-14T06:08:40.398 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3100 IOs, 1458849792 bytes 2024-09-14T06:08:40.480 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3200 IOs, 1460876288 bytes 2024-09-14T06:08:40.596 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3300 IOs, 1501522944 bytes 2024-09-14T06:08:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3400 IOs, 1562127360 bytes 2024-09-14T06:08:40.868 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3500 IOs, 1611919360 bytes 2024-09-14T06:08:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3600 IOs, 1633795072 bytes 2024-09-14T06:08:41.089 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3700 IOs, 1665466368 bytes 2024-09-14T06:08:41.250 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3800 IOs, 1747623936 bytes 2024-09-14T06:08:41.345 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:3900 IOs, 1759612928 bytes 2024-09-14T06:08:41.483 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4000 IOs, 1828544512 bytes 2024-09-14T06:08:41.589 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4100 IOs, 1847839744 bytes 2024-09-14T06:08:41.701 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4200 IOs, 1894456320 bytes 2024-09-14T06:08:41.853 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4300 IOs, 1958638592 bytes 2024-09-14T06:08:41.939 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4400 IOs, 1959232512 bytes 2024-09-14T06:08:42.061 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4500 IOs, 1997586432 bytes 2024-09-14T06:08:42.197 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4600 IOs, 2049625088 bytes 2024-09-14T06:08:42.328 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4700 IOs, 2101055488 bytes 2024-09-14T06:08:42.452 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4800 IOs, 2147249152 bytes 2024-09-14T06:08:42.600 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:4900 IOs, 2214901760 bytes 2024-09-14T06:08:42.710 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5000 IOs, 2261444608 bytes 2024-09-14T06:08:42.830 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5100 IOs, 2281655296 bytes 2024-09-14T06:08:42.947 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5200 IOs, 2323719168 bytes 2024-09-14T06:08:43.058 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5300 IOs, 2349088768 bytes 2024-09-14T06:08:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5400 IOs, 2376531968 bytes 2024-09-14T06:08:43.276 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:5500 IOs, 2392232960 bytes 2024-09-14T06:08:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=124 2024-09-14T06:08:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T06:08:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 124 -eq 124 ']' 2024-09-14T06:08:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:08:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_10 present 2024-09-14T06:08:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:08:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:08:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-14T06:08:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=present 2024-09-14T06:08:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:08:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:08:43.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:08:43.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_10 2024-09-14T06:08:43.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:08:43.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:08:43.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_10 2024-09-14T06:08:43.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-14T06:08:43.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:08:43.363 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id=127e55a66d97 2024-09-14T06:08:43.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:08:43.364 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:08:43.465 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_10 present 127e55a66d97 2024-09-14T06:08:43.465 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:08:43.465 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:08:43.465 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-14T06:08:43.465 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=present 2024-09-14T06:08:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id=127e55a66d97 2024-09-14T06:08:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:08:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:08:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_10 2024-09-14T06:08:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:08:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:08:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_10 2024-09-14T06:08:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-14T06:08:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:08:43.511 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id=127e55a66d97 2024-09-14T06:08:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 127e55a66d97 2024-09-14T06:08:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -z 127e55a66d97 -o 127e55a66d97 = 127e55a66d97 2024-09-14T06:08:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_state=present 2024-09-14T06:08:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test present = present 2024-09-14T06:08:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:08:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ disable_mirror cluster2 mirror image_10 2024-09-14T06:08:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:08:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:08:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-14T06:08:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_10 2024-09-14T06:09:00.770 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-14T06:09:00.769+0000 7fc6b05c7640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558ca7f97d40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-14T06:09:00.770 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-14T06:09:00.769+0000 7fc6b65d3640 -1 rbd::mirror::ImageReplayer: 0x558ca80e8280 [3/7fc24725-16ee-4027-a514-0ffde98759ee] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-14T06:09:07.658 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6699058705959726) > weight (0.25) 2024-09-14T06:09:07.658 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.628866454344235) > weight (0.25) 2024-09-14T06:09:07.658 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.8472771139927125) > weight (0.25) 2024-09-14T06:09:07.658 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.9265609655207233) > weight (0.25) 2024-09-14T06:09:07.658 INFO:tasks.rbd_mirror_thrash:waiting for 96 secs before thrashing 2024-09-14T06:09:14.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-14T06:09:14.289+0000 7f9b14f28640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55f03c3c2400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=194 2024-09-14T06:09:14.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-14T06:09:14.289+0000 7f9b14f28640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55f03c3c2400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=198, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=9d38de88-cffd-4132-b7e8-809c69f2745a, primary_snap_id=c2, last_copied_object_number=128, snap_seqs={194=18446744073709551614}] 2024-09-14T06:09:14.315 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:Mirroring disabled 2024-09-14T06:09:14.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2024-09-14T06:09:14.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:09:14.334 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:09:14.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-14T06:09:14.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local state=deleted 2024-09-14T06:09:14.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:09:14.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local s 2024-09-14T06:09:14.335 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:09:14.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_10 2024-09-14T06:09:14.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:09:14.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:09:14.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_10 2024-09-14T06:09:14.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:09:14.336 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-14T06:09:14.375 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-14T06:09:14.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ image_id= 2024-09-14T06:09:14.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-14T06:09:14.379 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0.1 2024-09-14T06:09:14.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2024-09-14T06:09:14.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:09:14.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:09:14.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-14T06:09:14.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local test_state=deleted 2024-09-14T06:09:14.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image_id= 2024-09-14T06:09:14.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_state=deleted 2024-09-14T06:09:14.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local current_image_id 2024-09-14T06:09:14.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ get_image_id cluster1 mirror image_10 2024-09-14T06:09:14.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:09:14.481 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:09:14.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local image=image_10 2024-09-14T06:09:14.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-14T06:09:14.482 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-14T06:09:14.517 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-14T06:09:14.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ current_image_id= 2024-09-14T06:09:14.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n '' 2024-09-14T06:09:14.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test deleted = deleted 2024-09-14T06:09:14.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:09:14.521 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ purge_snapshots cluster2 mirror image_10 2024-09-14T06:09:14.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:09:14.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:09:14.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-14T06:09:14.522 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster cluster2 snap purge mirror/image_10 2024-09-14T06:09:14.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image_retry cluster2 mirror image_10 2024-09-14T06:09:14.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:09:14.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:09:14.568 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-14T06:09:14.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-14T06:09:14.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:09:14.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ remove_image cluster2 mirror image_10 2024-09-14T06:09:14.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:09:14.569 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:09:14.570 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local image=image_10 2024-09-14T06:09:14.570 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2024-09-14T06:09:14.615 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2024-09-14T06:09:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-14T06:09:14.864 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:09:14.864 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ testlog 'TEST: check if removed images'\'' OMAP are removed' 2024-09-14T06:09:14.864 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ tee -a /tmp/tmp.sXArEbIrwv/rbd-mirror.test.log 2024-09-14T06:09:14.864 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ date '+%F %T' 2024-09-14T06:09:15.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo 2024-09-14 06:09:15 TEST: check if removed 'images'\''' OMAP are removed 2024-09-14T06:09:15.027 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:2024-09-14 06:09:15 TEST: check if removed images' OMAP are removed 2024-09-14T06:09:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_in_omap cluster1 mirror 2024-09-14T06:09:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:09:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:09:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring status_global 2024-09-14T06:09:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:09:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:09:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local obj_name=rbd_mirroring 2024-09-14T06:09:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local filter=status_global 2024-09-14T06:09:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-14T06:09:15.031 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:09:15.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T06:09:15.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring status_global 2024-09-14T06:09:15.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:09:15.032 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:09:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local obj_name=rbd_mirroring 2024-09-14T06:09:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local filter=status_global 2024-09-14T06:09:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2024-09-14T06:09:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:09:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:09:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -c status_global 2024-09-14T06:09:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local obj_name=rbd_mirroring 2024-09-14T06:09:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2024-09-14T06:09:15.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 0 = 0 2024-09-14T06:09:15.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=0 2024-09-14T06:09:15.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T06:09:15.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 0 -eq 0 ']' 2024-09-14T06:09:15.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:09:15.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring image_ 2024-09-14T06:09:15.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:09:15.120 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:09:15.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local obj_name=rbd_mirroring 2024-09-14T06:09:15.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local filter=image_ 2024-09-14T06:09:15.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-14T06:09:15.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:09:15.121 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T06:09:15.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring image_ 2024-09-14T06:09:15.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:09:15.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:09:15.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local obj_name=rbd_mirroring 2024-09-14T06:09:15.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local filter=image_ 2024-09-14T06:09:15.122 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2024-09-14T06:09:15.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:09:15.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:09:15.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local obj_name=rbd_mirroring 2024-09-14T06:09:15.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2024-09-14T06:09:15.123 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -c image_ 2024-09-14T06:09:15.159 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 0 = 0 2024-09-14T06:09:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=0 2024-09-14T06:09:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T06:09:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 0 -eq 0 ']' 2024-09-14T06:09:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:09:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirror_leader image_map 2024-09-14T06:09:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:09:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:09:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local obj_name=rbd_mirror_leader 2024-09-14T06:09:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local filter=image_map 2024-09-14T06:09:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-14T06:09:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:09:15.161 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T06:09:15.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2024-09-14T06:09:15.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:09:15.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:09:15.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local obj_name=rbd_mirror_leader 2024-09-14T06:09:15.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local filter=image_map 2024-09-14T06:09:15.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2024-09-14T06:09:15.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:09:15.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:09:15.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local obj_name=rbd_mirror_leader 2024-09-14T06:09:15.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2024-09-14T06:09:15.162 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -c image_map 2024-09-14T06:09:15.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 1 = 0 2024-09-14T06:09:15.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=1 2024-09-14T06:09:15.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T06:09:15.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 1 -eq 0 ']' 2024-09-14T06:09:15.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-14T06:09:15.196 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 1 2024-09-14T06:09:16.198 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T06:09:16.198 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2024-09-14T06:09:16.198 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:09:16.198 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:09:16.198 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local obj_name=rbd_mirror_leader 2024-09-14T06:09:16.198 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local filter=image_map 2024-09-14T06:09:16.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2024-09-14T06:09:16.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:09:16.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:09:16.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local obj_name=rbd_mirror_leader 2024-09-14T06:09:16.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2024-09-14T06:09:16.199 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -c image_map 2024-09-14T06:09:16.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 1 = 0 2024-09-14T06:09:16.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=1 2024-09-14T06:09:16.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T06:09:16.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 1 -eq 0 ']' 2024-09-14T06:09:16.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-14T06:09:16.235 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 2 2024-09-14T06:09:18.236 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T06:09:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2024-09-14T06:09:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:09:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:09:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local obj_name=rbd_mirror_leader 2024-09-14T06:09:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local filter=image_map 2024-09-14T06:09:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2024-09-14T06:09:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster1 2024-09-14T06:09:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:09:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local obj_name=rbd_mirror_leader 2024-09-14T06:09:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2024-09-14T06:09:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -c image_map 2024-09-14T06:09:18.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 0 = 0 2024-09-14T06:09:18.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=0 2024-09-14T06:09:18.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T06:09:18.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 0 -eq 0 ']' 2024-09-14T06:09:18.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:09:18.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_image_in_omap cluster2 mirror 2024-09-14T06:09:18.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:09:18.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:09:18.278 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring status_global 2024-09-14T06:09:18.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:09:18.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:09:18.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local obj_name=rbd_mirroring 2024-09-14T06:09:18.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local filter=status_global 2024-09-14T06:09:18.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-14T06:09:18.279 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:09:18.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T06:09:18.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring status_global 2024-09-14T06:09:18.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster2 2024-09-14T06:09:18.280 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:09:18.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local obj_name=rbd_mirroring 2024-09-14T06:09:18.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local filter=status_global 2024-09-14T06:09:18.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2024-09-14T06:09:18.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster2 2024-09-14T06:09:18.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:09:18.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local obj_name=rbd_mirroring 2024-09-14T06:09:18.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2024-09-14T06:09:18.281 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -c status_global 2024-09-14T06:09:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 0 = 0 2024-09-14T06:09:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=0 2024-09-14T06:09:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T06:09:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 0 -eq 0 ']' 2024-09-14T06:09:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:09:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring image_ 2024-09-14T06:09:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:09:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:09:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local obj_name=rbd_mirroring 2024-09-14T06:09:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local filter=image_ 2024-09-14T06:09:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-14T06:09:18.318 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:09:18.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T06:09:18.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring image_ 2024-09-14T06:09:18.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster2 2024-09-14T06:09:18.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:09:18.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local obj_name=rbd_mirroring 2024-09-14T06:09:18.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local filter=image_ 2024-09-14T06:09:18.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2024-09-14T06:09:18.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster2 2024-09-14T06:09:18.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:09:18.319 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local obj_name=rbd_mirroring 2024-09-14T06:09:18.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2024-09-14T06:09:18.320 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -c image_ 2024-09-14T06:09:18.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 0 = 0 2024-09-14T06:09:18.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=0 2024-09-14T06:09:18.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T06:09:18.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 0 -eq 0 ']' 2024-09-14T06:09:18.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:09:18.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirror_leader image_map 2024-09-14T06:09:18.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:09:18.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local pool=mirror 2024-09-14T06:09:18.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local obj_name=rbd_mirror_leader 2024-09-14T06:09:18.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local filter=image_map 2024-09-14T06:09:18.354 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-14T06:09:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ sleep 0 2024-09-14T06:09:18.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T06:09:18.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirror_leader image_map 2024-09-14T06:09:18.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster2 2024-09-14T06:09:18.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:09:18.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local obj_name=rbd_mirror_leader 2024-09-14T06:09:18.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local filter=image_map 2024-09-14T06:09:18.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ list_omap_keys cluster2 mirror rbd_mirror_leader 2024-09-14T06:09:18.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local cluster=cluster2 2024-09-14T06:09:18.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local pool=mirror 2024-09-14T06:09:18.356 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ local obj_name=rbd_mirror_leader 2024-09-14T06:09:18.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ grep -c image_map 2024-09-14T06:09:18.357 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirror_leader 2024-09-14T06:09:18.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test 0 = 0 2024-09-14T06:09:18.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ error_code=0 2024-09-14T06:09:18.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set -e 2024-09-14T06:09:18.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 0 -eq 0 ']' 2024-09-14T06:09:18.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 0 2024-09-14T06:09:18.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ cleanup 0 2024-09-14T06:09:18.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local error_code=0 2024-09-14T06:09:18.390 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ set +e 2024-09-14T06:09:18.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 0 -ne 0 ']' 2024-09-14T06:09:18.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -z '' ']' 2024-09-14T06:09:18.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster instance 2024-09-14T06:09:18.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ CEPH_ARGS= 2024-09-14T06:09:18.391 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ ceph --cluster cluster1 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-09-14T06:09:19.209 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:pool 'mirror' does not exist 2024-09-14T06:09:19.220 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ CEPH_ARGS= 2024-09-14T06:09:19.220 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ ceph --cluster cluster2 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-09-14T06:09:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:pool 'mirror' does not exist 2024-09-14T06:09:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ CEPH_ARGS= 2024-09-14T06:09:20.515 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ ceph --cluster cluster1 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-09-14T06:09:20.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-14T06:09:20.807+0000 7f213a2f4640 -1 rbd::mirror::RemotePoolPoller: 0x560b18e1e5b0 handle_mirror_peer_ping: failed to ping remote mirror peer: (2) No such file or directory 2024-09-14T06:09:20.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-14T06:09:20.807+0000 7f2139af3640 -1 rbd::mirror::RemotePoolPoller: 0x560b18e1e5b0 handle_mirror_peer_list: failed to retrieve mirror peers: (2) No such file or directory 2024-09-14T06:09:20.809 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-14T06:09:20.807+0000 7f2139af3640 -1 rbd::mirror::PoolReplayer: 0x560b18e31b00 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2024-09-14T06:09:20.810 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-14T06:09:20.808+0000 7f214730e640 -1 rbd::mirror::InstanceWatcher: 0x560b19d1e380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6636 object: (2) No such file or directory 2024-09-14T06:09:20.810 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-14T06:09:20.808+0000 7f214730e640 -1 rbd::mirror::InstanceWatcher: 0x560b1a20e000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6636 object: (2) No such file or directory 2024-09-14T06:09:20.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-14T06:09:20.808+0000 7f2140300640 -1 rbd::mirror::InstanceWatcher: 0x560b19d1e380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T06:09:20.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-14T06:09:20.808+0000 7f2140300640 -1 rbd::mirror::InstanceWatcher: 0x560b1a20e000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T06:09:20.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-14T06:09:20.809+0000 7f213a2f4640 -1 rbd::mirror::MirrorStatusUpdater 0x560b19d785a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T06:09:20.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-14T06:09:20.809+0000 7f214730e640 -1 rbd::mirror::NamespaceReplayer: 0x560b19d76000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T06:09:20.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-14T06:09:20.809+0000 7f2139af3640 -1 rbd::mirror::MirrorStatusUpdater 0x560b19d78360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T06:09:20.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-14T06:09:20.809+0000 7f214730e640 -1 rbd::mirror::NamespaceReplayer: 0x560b19d761a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T06:09:20.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-14T06:09:20.809+0000 7f2140b01640 -1 rbd::mirror::MirrorStatusUpdater 0x560b19d78000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T06:09:20.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-14T06:09:20.809+0000 7f2140b01640 -1 rbd::mirror::MirrorStatusUpdater 0x560b19d78120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T06:09:20.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-14T06:09:20.809+0000 7f2146b0d640 -1 rbd::mirror::NamespaceReplayer: 0x560b19d76000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T06:09:20.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-14T06:09:20.809+0000 7f2146b0d640 -1 rbd::mirror::NamespaceReplayer: 0x560b19d761a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T06:09:20.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-14T06:09:20.906+0000 7f2146b0d640 -1 rbd::mirror::LeaderWatcher: 0x560b1ab36c00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-14T06:09:20.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-14T06:09:20.907+0000 7f2146b0d640 -1 rbd::mirror::InstanceWatcher: 0x560b19d1e000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6636 object: (2) No such file or directory 2024-09-14T06:09:20.908 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-14T06:09:20.907+0000 7f2140300640 -1 rbd::mirror::InstanceWatcher: 0x560b19d1e000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T06:09:20.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-14T06:09:20.907+0000 7f213a2f4640 -1 rbd::mirror::MirrorStatusUpdater 0x560b1a9ba5a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T06:09:20.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-14T06:09:20.907+0000 7f2146b0d640 -1 rbd::mirror::NamespaceReplayer: 0x560b1a858000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T06:09:20.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-14T06:09:20.907+0000 7f2140b01640 -1 rbd::mirror::MirrorStatusUpdater 0x560b1a957560 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T06:09:20.910 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-14T06:09:20.908+0000 7f214730e640 -1 rbd::mirror::NamespaceReplayer: 0x560b1a858000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T06:09:21.217 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:pool 'mirror_parent' does not exist 2024-09-14T06:09:21.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ CEPH_ARGS= 2024-09-14T06:09:21.232 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ ceph --cluster cluster2 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-09-14T06:09:21.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-14T06:09:21.981+0000 7efbefd7f640 -1 rbd::mirror::RemotePoolPoller: 0x5626f54e15f0 handle_mirror_peer_ping: failed to ping remote mirror peer: (2) No such file or directory 2024-09-14T06:09:21.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-14T06:09:21.981+0000 7efbef57e640 -1 rbd::mirror::RemotePoolPoller: 0x5626f54e15f0 handle_mirror_peer_list: failed to retrieve mirror peers: (2) No such file or directory 2024-09-14T06:09:21.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-14T06:09:21.981+0000 7efbef57e640 -1 rbd::mirror::PoolReplayer: 0x5626f54f3b00 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2024-09-14T06:09:21.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-14T06:09:21.982+0000 7efbfc598640 -1 rbd::mirror::InstanceWatcher: 0x5626f6b9e000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6171 object: (2) No such file or directory 2024-09-14T06:09:21.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-14T06:09:21.982+0000 7efbfc598640 -1 rbd::mirror::InstanceWatcher: 0x5626f66b7c00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6171 object: (2) No such file or directory 2024-09-14T06:09:21.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-14T06:09:21.982+0000 7efbf658c640 -1 rbd::mirror::InstanceWatcher: 0x5626f6b9e000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T06:09:21.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-14T06:09:21.982+0000 7efbf658c640 -1 rbd::mirror::InstanceWatcher: 0x5626f66b7c00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T06:09:21.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-14T06:09:21.982+0000 7efbefd7f640 -1 rbd::mirror::MirrorStatusUpdater 0x5626f68feea0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T06:09:21.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-14T06:09:21.982+0000 7efbefd7f640 -1 rbd::mirror::MirrorStatusUpdater 0x5626f68fec60 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T06:09:21.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-14T06:09:21.982+0000 7efbfc598640 -1 rbd::mirror::NamespaceReplayer: 0x5626f6b6e000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T06:09:21.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-14T06:09:21.982+0000 7efbfc598640 -1 rbd::mirror::NamespaceReplayer: 0x5626f6b6e1a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T06:09:21.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-14T06:09:21.983+0000 7efbf658c640 -1 rbd::mirror::MirrorStatusUpdater 0x5626f68fe7e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T06:09:21.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-14T06:09:21.983+0000 7efbf658c640 -1 rbd::mirror::MirrorStatusUpdater 0x5626f68fe900 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T06:09:21.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-14T06:09:21.983+0000 7efbfcd99640 -1 rbd::mirror::NamespaceReplayer: 0x5626f6b6e000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T06:09:21.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-14T06:09:21.983+0000 7efbfcd99640 -1 rbd::mirror::NamespaceReplayer: 0x5626f6b6e1a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T06:09:22.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-14T06:09:22.249+0000 7efbfc598640 -1 rbd::mirror::LeaderWatcher: 0x5626f6618c00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-14T06:09:22.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-14T06:09:22.250+0000 7efbfc598640 -1 rbd::mirror::InstanceWatcher: 0x5626f66b6000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6171 object: (2) No such file or directory 2024-09-14T06:09:22.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-14T06:09:22.250+0000 7efbf5d8b640 -1 rbd::mirror::InstanceWatcher: 0x5626f66b6000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T06:09:22.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-14T06:09:22.250+0000 7efbef57e640 -1 rbd::mirror::MirrorStatusUpdater 0x5626f64726c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T06:09:22.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-14T06:09:22.250+0000 7efbfcd99640 -1 rbd::mirror::NamespaceReplayer: 0x5626f6f0a000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T06:09:22.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-14T06:09:22.251+0000 7efbf658c640 -1 rbd::mirror::MirrorStatusUpdater 0x5626f642f680 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T06:09:22.252 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-14T06:09:22.251+0000 7efbfc598640 -1 rbd::mirror::NamespaceReplayer: 0x5626f6f0a000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T06:09:22.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-14T06:09:22.254+0000 7efbe8570640 -1 rbd::mirror::InstanceWatcher: 0x5626f66b7880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T06:09:22.402 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:pool 'mirror_parent' does not exist 2024-09-14T06:09:22.413 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-09-14T06:09:22.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stop_mirrors cluster1 2024-09-14T06:09:22.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1 2024-09-14T06:09:22.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local sig= 2024-09-14T06:09:22.414 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ seq 0 3 2024-09-14T06:09:22.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:09:22.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stop_mirror cluster1:0 '' 2024-09-14T06:09:22.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1:0 2024-09-14T06:09:22.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local sig= 2024-09-14T06:09:22.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 1 2024-09-14T06:09:22.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 2024-09-14T06:09:22.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:09:22.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stop_mirror cluster1:1 '' 2024-09-14T06:09:22.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1:1 2024-09-14T06:09:22.418 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local sig= 2024-09-14T06:09:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 1 2024-09-14T06:09:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 2024-09-14T06:09:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:09:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stop_mirror cluster1:2 '' 2024-09-14T06:09:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1:2 2024-09-14T06:09:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local sig= 2024-09-14T06:09:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 1 2024-09-14T06:09:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 2024-09-14T06:09:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:09:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stop_mirror cluster1:3 '' 2024-09-14T06:09:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster1:3 2024-09-14T06:09:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local sig= 2024-09-14T06:09:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 1 2024-09-14T06:09:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 2024-09-14T06:09:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-09-14T06:09:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stop_mirrors cluster2 2024-09-14T06:09:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2 2024-09-14T06:09:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local sig= 2024-09-14T06:09:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:++ seq 0 3 2024-09-14T06:09:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:09:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stop_mirror cluster2:0 '' 2024-09-14T06:09:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2:0 2024-09-14T06:09:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local sig= 2024-09-14T06:09:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 1 2024-09-14T06:09:22.421 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 2024-09-14T06:09:22.421 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:09:22.421 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stop_mirror cluster2:1 '' 2024-09-14T06:09:22.421 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2:1 2024-09-14T06:09:22.421 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local sig= 2024-09-14T06:09:22.421 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 1 2024-09-14T06:09:22.421 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 2024-09-14T06:09:22.421 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:09:22.421 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stop_mirror cluster2:2 '' 2024-09-14T06:09:22.421 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2:2 2024-09-14T06:09:22.421 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local sig= 2024-09-14T06:09:22.421 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 1 2024-09-14T06:09:22.421 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 2024-09-14T06:09:22.422 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-14T06:09:22.422 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ stop_mirror cluster2:3 '' 2024-09-14T06:09:22.422 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local cluster=cluster2:3 2024-09-14T06:09:22.422 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ local sig= 2024-09-14T06:09:22.422 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test -n 1 2024-09-14T06:09:22.422 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ return 2024-09-14T06:09:22.422 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' -z 1 ']' 2024-09-14T06:09:22.422 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ test '' = /tmp/tmp.sXArEbIrwv 2024-09-14T06:09:22.422 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ rm -Rf /tmp/tmp.sXArEbIrwv 2024-09-14T06:09:22.448 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stdout:OK 2024-09-14T06:09:22.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ '[' 0 -eq 0 ']' 2024-09-14T06:09:22.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ echo OK 2024-09-14T06:09:22.449 INFO:tasks.workunit.cluster1.client.mirror.smithi191.stderr:+ exit 0 2024-09-14T06:09:22.454 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-14T06:09:22.454 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp 2024-09-14T06:09:22.577 INFO:tasks.workunit:Stopping ['rbd/rbd_mirror_stress.sh'] on cluster1.client.mirror... 2024-09-14T06:09:22.577 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.cluster1.client.mirror /home/ubuntu/cephtest/clone.cluster1.client.mirror 2024-09-14T06:09:22.836 DEBUG:teuthology.parallel:result is None 2024-09-14T06:09:22.837 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-09-14T06:09:22.861 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-09-14T06:09:22.861 DEBUG:teuthology.orchestra.run.smithi191:> rmdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-14T06:09:22.924 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-09-14T06:09:22.924 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2024-09-14T06:09:22.934 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2024-09-14T06:09:22.935 INFO:tasks.rbd_mirror_thrash:stat['kill'] = 16 2024-09-14T06:09:22.935 INFO:tasks.rbd_mirror_thrash:done joining 2024-09-14T06:09:22.935 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2024-09-14T06:09:23.080 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2024-09-14T06:09:23.080 INFO:tasks.rbd_mirror_thrash:stat['kill'] = 16 2024-09-14T06:09:23.080 INFO:tasks.rbd_mirror_thrash:done joining 2024-09-14T06:09:23.080 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-14T06:09:23.218 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.3:waiting for process to exit 2024-09-14T06:09:23.218 INFO:teuthology.orchestra.run:waiting for 300 2024-09-14T06:09:23.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.218+0000 7ff4cd588640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 42602) UID: 1000 2024-09-14T06:09:23.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.218+0000 7ff4ccd87640 -1 rbd::mirror::LeaderWatcher: 0x55b42ee43200 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-14T06:09:23.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.219+0000 7ff4ccd87640 -1 rbd::mirror::InstanceWatcher: 0x55b42ec22380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4538 object: (2) No such file or directory 2024-09-14T06:09:23.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.219+0000 7ff4ccd87640 -1 rbd::mirror::InstanceWatcher: 0x55b42ec22700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4538 object: (2) No such file or directory 2024-09-14T06:09:23.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.219+0000 7ff4ccd87640 -1 rbd::mirror::InstanceWatcher: 0x55b42ec22a80 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4541 object: (2) No such file or directory 2024-09-14T06:09:23.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.219+0000 7ff4c657a640 -1 rbd::mirror::InstanceWatcher: 0x55b42ec22380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T06:09:23.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.219+0000 7ff4c657a640 -1 rbd::mirror::InstanceWatcher: 0x55b42ec22700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T06:09:23.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.219+0000 7ff4b855e640 -1 rbd::mirror::InstanceWatcher: 0x55b42ec22a80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T06:09:23.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.219+0000 7ff4bf56c640 -1 rbd::mirror::MirrorStatusUpdater 0x55b42ec70360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T06:09:23.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.219+0000 7ff4bf56c640 -1 rbd::mirror::MirrorStatusUpdater 0x55b42ec705a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T06:09:23.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.219+0000 7ff4cc586640 -1 rbd::mirror::NamespaceReplayer: 0x55b42ec6e000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T06:09:23.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.219+0000 7ff4b2552640 -1 rbd::mirror::MirrorStatusUpdater 0x55b42eea67e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T06:09:23.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.219+0000 7ff4cc586640 -1 rbd::mirror::NamespaceReplayer: 0x55b42ec6e1a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T06:09:23.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.219+0000 7ff4cc586640 -1 rbd::mirror::NamespaceReplayer: 0x55b42e8b21a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T06:09:23.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.220+0000 7ff4c657a640 -1 rbd::mirror::MirrorStatusUpdater 0x55b42ec70000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T06:09:23.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.220+0000 7ff4c657a640 -1 rbd::mirror::MirrorStatusUpdater 0x55b42ec70120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T06:09:23.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.220+0000 7ff4ccd87640 -1 rbd::mirror::NamespaceReplayer: 0x55b42ec6e000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T06:09:23.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.220+0000 7ff4b8d5f640 -1 rbd::mirror::MirrorStatusUpdater 0x55b42ee519e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T06:09:23.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.220+0000 7ff4ccd87640 -1 rbd::mirror::NamespaceReplayer: 0x55b42ec6e1a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T06:09:23.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.220+0000 7ff4ccd87640 -1 rbd::mirror::NamespaceReplayer: 0x55b42e8b21a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T06:09:23.222 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.222+0000 7ff4ccd87640 -1 rbd::mirror::LeaderWatcher: 0x55b42eb7ec00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-14T06:09:23.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.222+0000 7ff4cc586640 -1 rbd::mirror::InstanceWatcher: 0x55b42ec22000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4538 object: (2) No such file or directory 2024-09-14T06:09:23.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.222+0000 7ff4c5d79640 -1 rbd::mirror::InstanceWatcher: 0x55b42ec22000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T06:09:23.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.223+0000 7ff4bfd6d640 -1 rbd::mirror::MirrorStatusUpdater 0x55b42e9887e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T06:09:23.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.223+0000 7ff4cc586640 -1 rbd::mirror::NamespaceReplayer: 0x55b42e8b2340 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T06:09:23.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.223+0000 7ff4c657a640 -1 rbd::mirror::MirrorStatusUpdater 0x55b42e97d440 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T06:09:23.224 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.223+0000 7ff4ccd87640 -1 rbd::mirror::NamespaceReplayer: 0x55b42e8b2340 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T06:09:23.235 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Stopped 2024-09-14T06:09:23.236 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-14T06:09:23.335 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.2:waiting for process to exit 2024-09-14T06:09:23.335 INFO:teuthology.orchestra.run:waiting for 300 2024-09-14T06:09:23.335 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-14T06:09:23.334+0000 7f2147b0f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 70541) UID: 1000 2024-09-14T06:09:23.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-14T06:09:23.335+0000 7f214730e640 -1 rbd::mirror::LeaderWatcher: 0x560b19f27200 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-14T06:09:23.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-14T06:09:23.335+0000 7f214730e640 -1 rbd::mirror::InstanceWatcher: 0x560b19d1fc00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6639 object: (2) No such file or directory 2024-09-14T06:09:23.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-14T06:09:23.335+0000 7f21332e6640 -1 rbd::mirror::InstanceWatcher: 0x560b19d1fc00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T06:09:23.337 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-14T06:09:23.336+0000 7f212c2d8640 -1 rbd::mirror::MirrorStatusUpdater 0x560b19f606c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T06:09:23.337 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-14T06:09:23.336+0000 7f214730e640 -1 rbd::mirror::NamespaceReplayer: 0x560b19c241a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T06:09:23.337 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-14T06:09:23.336+0000 7f2132ae5640 -1 rbd::mirror::MirrorStatusUpdater 0x560b19f5d7a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T06:09:23.337 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi191.stderr:2024-09-14T06:09:23.336+0000 7f2146b0d640 -1 rbd::mirror::NamespaceReplayer: 0x560b19c241a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T06:09:23.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Stopped 2024-09-14T06:09:23.348 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-14T06:09:23.427 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.1:waiting for process to exit 2024-09-14T06:09:23.427 INFO:teuthology.orchestra.run:waiting for 300 2024-09-14T06:09:23.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.357+0000 7f9b21f42640 -1 rbd::mirror::LeaderWatcher: 0x55f0392ccc00 handle_notify_heartbeat: error notifying heartbeat: (2) No such file or directory, releasing leader 2024-09-14T06:09:23.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.357+0000 7f9b1af34640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55f0398238c0 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-14T06:09:23.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.357+0000 7f9b1af34640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55f039348b40 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-14T06:09:23.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.357+0000 7f9b1af34640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55f039348240 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-14T06:09:23.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.357+0000 7f9b1b735640 -1 rbd::mirror::PoolWatcher: 0x55f039dfe960 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-14T06:09:23.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.357+0000 7f9b1af34640 -1 rbd::mirror::PoolWatcher: 0x55f039dfe000 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-14T06:09:23.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.357+0000 7f9b1af34640 -1 rbd::mirror::PoolWatcher: 0x55f039dfeb40 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-14T06:09:23.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.357+0000 7f9b14f28640 -1 rbd::mirror::PoolWatcher: 0x55f039dfef00 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-14T06:09:23.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.357+0000 7f9b14f28640 -1 rbd::mirror::PoolWatcher: 0x55f039dfe1e0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-14T06:09:23.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.357+0000 7f9b14f28640 -1 rbd::mirror::PoolWatcher: 0x55f039dfed20 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-14T06:09:23.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.358+0000 7f9b21741640 -1 rbd::mirror::LeaderWatcher: 0x55f0392ccc00 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-09-14T06:09:23.430 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.358+0000 7f9b21741640 -1 rbd::mirror::LeaderWatcher: 0x55f0392ccc00 handle_release_leader_lock: error releasing lock: (2) No such file or directory 2024-09-14T06:09:23.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:09:23.427+0000 7f944fc4c640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 71283) UID: 1000 2024-09-14T06:09:23.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:09:23.428+0000 7f944f44b640 -1 rbd::mirror::LeaderWatcher: 0x5601f7e3f200 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-14T06:09:23.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:09:23.428+0000 7f944ec4a640 -1 rbd::mirror::InstanceWatcher: 0x5601f7c2c380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6700 object: (2) No such file or directory 2024-09-14T06:09:23.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:09:23.428+0000 7f944ec4a640 -1 rbd::mirror::InstanceWatcher: 0x5601f7c2ca80 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6700 object: (2) No such file or directory 2024-09-14T06:09:23.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:09:23.428+0000 7f944ec4a640 -1 rbd::mirror::InstanceWatcher: 0x5601f7c2c700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6702 object: (2) No such file or directory 2024-09-14T06:09:23.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:09:23.428+0000 7f9448c3e640 -1 rbd::mirror::InstanceWatcher: 0x5601f7c2c380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T06:09:23.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:09:23.428+0000 7f9448c3e640 -1 rbd::mirror::InstanceWatcher: 0x5601f7c2ca80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T06:09:23.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:09:23.428+0000 7f943ac22640 -1 rbd::mirror::InstanceWatcher: 0x5601f7c2c700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T06:09:23.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:09:23.429+0000 7f9442431640 -1 rbd::mirror::MirrorStatusUpdater 0x5601f7c9e5a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T06:09:23.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:09:23.429+0000 7f9442431640 -1 rbd::mirror::MirrorStatusUpdater 0x5601f7c9e360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T06:09:23.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:09:23.429+0000 7f944ec4a640 -1 rbd::mirror::NamespaceReplayer: 0x5601f7c9c000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T06:09:23.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:09:23.429+0000 7f944ec4a640 -1 rbd::mirror::NamespaceReplayer: 0x5601f7c9c1a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T06:09:23.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:09:23.429+0000 7f9434415640 -1 rbd::mirror::MirrorStatusUpdater 0x5601f7ec06c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T06:09:23.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:09:23.429+0000 7f944ec4a640 -1 rbd::mirror::NamespaceReplayer: 0x5601f7c9c4e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T06:09:23.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:09:23.429+0000 7f944843d640 -1 rbd::mirror::MirrorStatusUpdater 0x5601f7c9e000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T06:09:23.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:09:23.429+0000 7f944843d640 -1 rbd::mirror::MirrorStatusUpdater 0x5601f7c9e120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T06:09:23.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:09:23.429+0000 7f943b423640 -1 rbd::mirror::MirrorStatusUpdater 0x5601f7eb1b00 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T06:09:23.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:09:23.429+0000 7f944f44b640 -1 rbd::mirror::NamespaceReplayer: 0x5601f7c9c000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T06:09:23.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:09:23.429+0000 7f944f44b640 -1 rbd::mirror::NamespaceReplayer: 0x5601f7c9c1a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T06:09:23.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:09:23.429+0000 7f944f44b640 -1 rbd::mirror::NamespaceReplayer: 0x5601f7c9c4e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T06:09:23.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:09:23.431+0000 7f944ec4a640 -1 rbd::mirror::LeaderWatcher: 0x5601f7b80c00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-14T06:09:23.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:09:23.431+0000 7f944ec4a640 -1 rbd::mirror::InstanceWatcher: 0x5601f7c2c000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6700 object: (2) No such file or directory 2024-09-14T06:09:23.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:09:23.432+0000 7f9448c3e640 -1 rbd::mirror::InstanceWatcher: 0x5601f7c2c000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T06:09:23.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:09:23.432+0000 7f9441c30640 -1 rbd::mirror::MirrorStatusUpdater 0x5601f79ea6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T06:09:23.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:09:23.432+0000 7f944ec4a640 -1 rbd::mirror::NamespaceReplayer: 0x5601f7827d40 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T06:09:23.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:09:23.432+0000 7f944843d640 -1 rbd::mirror::MirrorStatusUpdater 0x5601f7983440 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T06:09:23.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi191.stderr:2024-09-14T06:09:23.432+0000 7f944f44b640 -1 rbd::mirror::NamespaceReplayer: 0x5601f7827d40 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T06:09:23.444 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Stopped 2024-09-14T06:09:23.445 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-14T06:09:23.485 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.0:waiting for process to exit 2024-09-14T06:09:23.485 INFO:teuthology.orchestra.run:waiting for 300 2024-09-14T06:09:23.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.485+0000 7fc6b75d5640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 69244) UID: 1000 2024-09-14T06:09:23.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.486+0000 7fc6afdc6640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x558ca7e67440 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-14T06:09:23.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.486+0000 7fc6a25ab640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x558ca7c3e480 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-14T06:09:23.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.486+0000 7fc6afdc6640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x558ca88ee240 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-14T06:09:23.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.486+0000 7fc6a2dac640 -1 rbd::mirror::PoolWatcher: 0x558ca6b20f00 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-14T06:09:23.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.486+0000 7fc6b05c7640 -1 rbd::mirror::PoolWatcher: 0x558ca7e58f00 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-14T06:09:23.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.486+0000 7fc6b05c7640 -1 rbd::mirror::PoolWatcher: 0x558ca80865a0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-14T06:09:23.488 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.486+0000 7fc6a95b9640 -1 rbd::mirror::PoolWatcher: 0x558ca80861e0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-14T06:09:23.488 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.486+0000 7fc69bd9e640 -1 rbd::mirror::PoolWatcher: 0x558ca6b210e0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-14T06:09:23.488 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.486+0000 7fc6a9dba640 -1 rbd::mirror::PoolWatcher: 0x558ca8086000 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-14T06:09:23.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.487+0000 7fc6b6dd4640 -1 rbd::mirror::LeaderWatcher: 0x558ca7c57200 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-09-14T06:09:23.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.487+0000 7fc6b6dd4640 -1 rbd::mirror::InstanceWatcher: 0x558ca7a50380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6525 object: (2) No such file or directory 2024-09-14T06:09:23.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.487+0000 7fc6b6dd4640 -1 rbd::mirror::InstanceWatcher: 0x558ca7f24000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6525 object: (2) No such file or directory 2024-09-14T06:09:23.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.487+0000 7fc6b6dd4640 -1 rbd::mirror::LeaderWatcher: 0x558ca7c57200 handle_shut_down_leader_lock: error shutting down leader lock: (2) No such file or directory 2024-09-14T06:09:23.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.487+0000 7fc6b65d3640 -1 rbd::mirror::LeaderWatcher: 0x558ca7c57200 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-14T06:09:23.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.487+0000 7fc6b05c7640 -1 rbd::mirror::InstanceWatcher: 0x558ca7a50380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T06:09:23.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.487+0000 7fc6b05c7640 -1 rbd::mirror::InstanceWatcher: 0x558ca7f24000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T06:09:23.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.487+0000 7fc6b65d3640 -1 rbd::mirror::InstanceWatcher: 0x558ca7a51c00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6528 object: (2) No such file or directory 2024-09-14T06:09:23.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.487+0000 7fc6a9dba640 -1 rbd::mirror::MirrorStatusUpdater 0x558ca7aa25a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T06:09:23.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.487+0000 7fc6a9dba640 -1 rbd::mirror::MirrorStatusUpdater 0x558ca7aa2360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T06:09:23.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.487+0000 7fc6b6dd4640 -1 rbd::mirror::NamespaceReplayer: 0x558ca7aa0000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T06:09:23.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.487+0000 7fc6b6dd4640 -1 rbd::mirror::NamespaceReplayer: 0x558ca7aa01a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T06:09:23.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.487+0000 7fc6a2dac640 -1 rbd::mirror::InstanceWatcher: 0x558ca7a51c00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T06:09:23.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.488+0000 7fc6afdc6640 -1 rbd::mirror::MirrorStatusUpdater 0x558ca7aa2000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T06:09:23.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.488+0000 7fc6afdc6640 -1 rbd::mirror::MirrorStatusUpdater 0x558ca7aa2120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T06:09:23.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.488+0000 7fc6b65d3640 -1 rbd::mirror::NamespaceReplayer: 0x558ca7aa0000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T06:09:23.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.488+0000 7fc6b65d3640 -1 rbd::mirror::NamespaceReplayer: 0x558ca7aa01a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T06:09:23.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.488+0000 7fc69c59f640 -1 rbd::mirror::MirrorStatusUpdater 0x558ca7aa26c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T06:09:23.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.488+0000 7fc6b6dd4640 -1 rbd::mirror::NamespaceReplayer: 0x558ca7aa04e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T06:09:23.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.488+0000 7fc6a25ab640 -1 rbd::mirror::MirrorStatusUpdater 0x558ca7c737a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T06:09:23.491 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.488+0000 7fc6b65d3640 -1 rbd::mirror::NamespaceReplayer: 0x558ca7aa04e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T06:09:23.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.490+0000 7fc6b05c7640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x558ca7e67680 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-14T06:09:23.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.490+0000 7fc6afdc6640 -1 rbd::mirror::PoolWatcher: 0x558ca80863c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-14T06:09:23.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.490+0000 7fc6a95b9640 -1 rbd::mirror::PoolWatcher: 0x558ca8086780 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-14T06:09:23.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.491+0000 7fc6b65d3640 -1 rbd::mirror::LeaderWatcher: 0x558ca8880c00 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-09-14T06:09:23.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.491+0000 7fc6b65d3640 -1 rbd::mirror::LeaderWatcher: 0x558ca8880c00 handle_shut_down_leader_lock: error shutting down leader lock: (2) No such file or directory 2024-09-14T06:09:23.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.491+0000 7fc6b6dd4640 -1 rbd::mirror::LeaderWatcher: 0x558ca8880c00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-14T06:09:23.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.491+0000 7fc6b6dd4640 -1 rbd::mirror::InstanceWatcher: 0x558ca7a50000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6525 object: (2) No such file or directory 2024-09-14T06:09:23.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.491+0000 7fc6b05c7640 -1 rbd::mirror::InstanceWatcher: 0x558ca7a50000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T06:09:23.493 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.492+0000 7fc6a9dba640 -1 rbd::mirror::MirrorStatusUpdater 0x558ca86cc6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T06:09:23.494 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.492+0000 7fc6b65d3640 -1 rbd::mirror::NamespaceReplayer: 0x558ca7933a00 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T06:09:23.494 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.492+0000 7fc6afdc6640 -1 rbd::mirror::MirrorStatusUpdater 0x558ca8669440 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T06:09:23.494 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.492+0000 7fc6b6dd4640 -1 rbd::mirror::NamespaceReplayer: 0x558ca7933a00 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T06:09:23.505 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Stopped 2024-09-14T06:09:23.505 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-14T06:09:23.633 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.3:waiting for process to exit 2024-09-14T06:09:23.634 INFO:teuthology.orchestra.run:waiting for 300 2024-09-14T06:09:23.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.633+0000 7f9b22743640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 46764) UID: 1000 2024-09-14T06:09:23.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.634+0000 7f9b21741640 -1 rbd::mirror::InstanceWatcher: 0x55f039374380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4734 object: (2) No such file or directory 2024-09-14T06:09:23.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.634+0000 7f9b21741640 -1 rbd::mirror::InstanceWatcher: 0x55f039bfbc00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4734 object: (2) No such file or directory 2024-09-14T06:09:23.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.634+0000 7f9b21741640 -1 rbd::mirror::InstanceWatcher: 0x55f039374700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4736 object: (2) No such file or directory 2024-09-14T06:09:23.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.634+0000 7f9b1af34640 -1 rbd::mirror::InstanceWatcher: 0x55f039374380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T06:09:23.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.634+0000 7f9b1af34640 -1 rbd::mirror::InstanceWatcher: 0x55f039bfbc00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T06:09:23.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.634+0000 7f9b0df1a640 -1 rbd::mirror::InstanceWatcher: 0x55f039374700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T06:09:23.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.635+0000 7f9b14f28640 -1 rbd::mirror::MirrorStatusUpdater 0x55f0393fc360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T06:09:23.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.635+0000 7f9b14f28640 -1 rbd::mirror::MirrorStatusUpdater 0x55f0393fc5a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T06:09:23.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.635+0000 7f9b21f42640 -1 rbd::mirror::NamespaceReplayer: 0x55f0393fa000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T06:09:23.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.635+0000 7f9b0770d640 -1 rbd::mirror::MirrorStatusUpdater 0x55f0396087e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T06:09:23.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.635+0000 7f9b21f42640 -1 rbd::mirror::NamespaceReplayer: 0x55f0393fa1a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T06:09:23.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.635+0000 7f9b21f42640 -1 rbd::mirror::NamespaceReplayer: 0x55f039882000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T06:09:23.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.635+0000 7f9b1af34640 -1 rbd::mirror::MirrorStatusUpdater 0x55f0393fc000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T06:09:23.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.635+0000 7f9b1af34640 -1 rbd::mirror::MirrorStatusUpdater 0x55f0393fc120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T06:09:23.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.635+0000 7f9b21f42640 -1 rbd::mirror::NamespaceReplayer: 0x55f0393fa000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T06:09:23.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.635+0000 7f9b21741640 -1 rbd::mirror::NamespaceReplayer: 0x55f0393fa1a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T06:09:23.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.637+0000 7f9b21741640 -1 rbd::mirror::LeaderWatcher: 0x55f0392ccc00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-14T06:09:23.638 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.637+0000 7f9b21741640 -1 rbd::mirror::InstanceWatcher: 0x55f039374000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4734 object: (2) No such file or directory 2024-09-14T06:09:23.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.638+0000 7f9b1b735640 -1 rbd::mirror::InstanceWatcher: 0x55f039374000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T06:09:23.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.638+0000 7f9b14727640 -1 rbd::mirror::MirrorStatusUpdater 0x55f039d8c7e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T06:09:23.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.638+0000 7f9b21741640 -1 rbd::mirror::NamespaceReplayer: 0x55f03820c9c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T06:09:23.639 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.638+0000 7f9b1af34640 -1 rbd::mirror::MirrorStatusUpdater 0x55f039d57440 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T06:09:23.640 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi191.stderr:2024-09-14T06:09:23.638+0000 7f9b21f42640 -1 rbd::mirror::NamespaceReplayer: 0x55f03820c9c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T06:09:23.657 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Stopped 2024-09-14T06:09:23.657 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-14T06:09:23.798 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.2:waiting for process to exit 2024-09-14T06:09:23.799 INFO:teuthology.orchestra.run:waiting for 300 2024-09-14T06:09:23.799 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi191.stderr:2024-09-14T06:09:23.798+0000 7efbfd59a640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 71640) UID: 1000 2024-09-14T06:09:23.809 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Stopped 2024-09-14T06:09:23.809 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-14T06:09:23.843 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.1:waiting for process to exit 2024-09-14T06:09:23.843 INFO:teuthology.orchestra.run:waiting for 300 2024-09-14T06:09:23.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:2024-09-14T06:09:23.849+0000 7fb4104e2640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 70168) UID: 1000 2024-09-14T06:09:23.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:2024-09-14T06:09:23.850+0000 7fb40f4e0640 -1 rbd::mirror::InstanceWatcher: 0x5564beaedc00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6093 object: (2) No such file or directory 2024-09-14T06:09:23.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:2024-09-14T06:09:23.850+0000 7fb40fce1640 -1 rbd::mirror::InstanceWatcher: 0x5564befc8000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6093 object: (2) No such file or directory 2024-09-14T06:09:23.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:2024-09-14T06:09:23.850+0000 7fb3fb4b8640 -1 rbd::mirror::InstanceWatcher: 0x5564beaed880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T06:09:23.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:2024-09-14T06:09:23.850+0000 7fb408cd3640 -1 rbd::mirror::InstanceWatcher: 0x5564beaedc00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T06:09:23.851 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:2024-09-14T06:09:23.850+0000 7fb4094d4640 -1 rbd::mirror::InstanceWatcher: 0x5564befc8000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T06:09:23.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:2024-09-14T06:09:23.850+0000 7fb4024c6640 -1 rbd::mirror::MirrorStatusUpdater 0x5564bed28ea0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T06:09:23.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:2024-09-14T06:09:23.851+0000 7fb4024c6640 -1 rbd::mirror::MirrorStatusUpdater 0x5564bed28c60 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T06:09:23.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:2024-09-14T06:09:23.851+0000 7fb40fce1640 -1 rbd::mirror::NamespaceReplayer: 0x5564befa2000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T06:09:23.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:2024-09-14T06:09:23.851+0000 7fb40fce1640 -1 rbd::mirror::NamespaceReplayer: 0x5564befa21a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T06:09:23.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:2024-09-14T06:09:23.851+0000 7fb408cd3640 -1 rbd::mirror::MirrorStatusUpdater 0x5564bed287e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T06:09:23.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:2024-09-14T06:09:23.851+0000 7fb408cd3640 -1 rbd::mirror::MirrorStatusUpdater 0x5564bed28900 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T06:09:23.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:2024-09-14T06:09:23.851+0000 7fb40f4e0640 -1 rbd::mirror::NamespaceReplayer: 0x5564befa2000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T06:09:23.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:2024-09-14T06:09:23.851+0000 7fb40f4e0640 -1 rbd::mirror::NamespaceReplayer: 0x5564befa21a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T06:09:23.853 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:2024-09-14T06:09:23.852+0000 7fb40f4e0640 -1 rbd::mirror::LeaderWatcher: 0x5564bf91ec00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-14T06:09:23.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:2024-09-14T06:09:23.853+0000 7fb40f4e0640 -1 rbd::mirror::InstanceWatcher: 0x5564beaec000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6093 object: (2) No such file or directory 2024-09-14T06:09:23.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:2024-09-14T06:09:23.853+0000 7fb4094d4640 -1 rbd::mirror::InstanceWatcher: 0x5564beaec000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T06:09:23.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:2024-09-14T06:09:23.853+0000 7fb402cc7640 -1 rbd::mirror::MirrorStatusUpdater 0x5564bf7746c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T06:09:23.854 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:2024-09-14T06:09:23.853+0000 7fb40fce1640 -1 rbd::mirror::NamespaceReplayer: 0x5564bf61e000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T06:09:23.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:2024-09-14T06:09:23.853+0000 7fb408cd3640 -1 rbd::mirror::MirrorStatusUpdater 0x5564bf713440 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T06:09:23.855 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi191.stderr:2024-09-14T06:09:23.853+0000 7fb40f4e0640 -1 rbd::mirror::NamespaceReplayer: 0x5564bf61e000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T06:09:23.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Stopped 2024-09-14T06:09:23.868 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-14T06:09:23.907 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.0:waiting for process to exit 2024-09-14T06:09:23.907 INFO:teuthology.orchestra.run:waiting for 300 2024-09-14T06:09:23.909 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.908+0000 7f7632f48640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 70858) UID: 1000 2024-09-14T06:09:23.910 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.909+0000 7f761df1e640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x556da1586240 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-14T06:09:23.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.909+0000 7f761e71f640 -1 rbd::mirror::PoolWatcher: 0x556da1ac54a0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-14T06:09:23.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.909+0000 7f762bf3a640 -1 rbd::mirror::InstanceWatcher: 0x556da15ca380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T06:09:23.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.909+0000 7f7617711640 -1 rbd::mirror::PoolWatcher: 0x556da1ac5860 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-14T06:09:23.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.909+0000 7f762b739640 -1 rbd::mirror::InstanceWatcher: 0x556da15ca700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T06:09:23.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.909+0000 7f762572d640 -1 rbd::mirror::MirrorStatusUpdater 0x556da16467e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T06:09:23.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.909+0000 7f762572d640 -1 rbd::mirror::MirrorStatusUpdater 0x556da1646360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T06:09:23.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.909+0000 7f7632747640 -1 rbd::mirror::NamespaceReplayer: 0x556da1644000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T06:09:23.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.909+0000 7f7632747640 -1 rbd::mirror::NamespaceReplayer: 0x556da16441a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T06:09:23.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.909+0000 7f7632747640 -1 rbd::mirror::LeaderWatcher: 0x556da1833200 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-09-14T06:09:23.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.909+0000 7f7632747640 -1 rbd::mirror::LeaderWatcher: 0x556da1833200 handle_shut_down_leader_lock: error shutting down leader lock: (2) No such file or directory 2024-09-14T06:09:23.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.910+0000 7f7631f46640 -1 rbd::mirror::LeaderWatcher: 0x556da1833200 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-14T06:09:23.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.910+0000 7f7631f46640 -1 rbd::mirror::InstanceWatcher: 0x556da1a9c000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6128 object: (2) No such file or directory 2024-09-14T06:09:23.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.910+0000 7f761e71f640 -1 rbd::mirror::InstanceWatcher: 0x556da1a9c000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T06:09:23.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.910+0000 7f7617f12640 -1 rbd::mirror::MirrorStatusUpdater 0x556da185a360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T06:09:23.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.910+0000 7f7632747640 -1 rbd::mirror::NamespaceReplayer: 0x556da16444e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T06:09:23.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.911+0000 7f761df1e640 -1 rbd::mirror::MirrorStatusUpdater 0x556da17e99e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T06:09:23.913 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.911+0000 7f7631f46640 -1 rbd::mirror::NamespaceReplayer: 0x556da16444e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-14T06:09:23.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.913+0000 7f762bf3a640 -1 rbd::mirror::InstanceWatcher: 0x556da15ca000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-14T06:09:23.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.913+0000 7f7624f2c640 -1 rbd::mirror::MirrorStatusUpdater 0x556da134c6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-14T06:09:23.914 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi191.stderr:2024-09-14T06:09:23.913+0000 7f7632747640 -1 rbd::mirror::NamespaceReplayer: 0x556da1d8a000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-14T06:09:23.925 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Stopped 2024-09-14T06:09:23.925 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-14T06:09:23.992 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-09-14T06:09:23.993 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-14T06:09:24.289 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T06:09:24.290 INFO:teuthology.orchestra.run.smithi191.stderr:dumped all 2024-09-14T06:09:24.302 INFO:teuthology.orchestra.run.smithi191.stdout:{"pg_ready":true,"pg_map":{"version":1589,"stamp":"2024-09-14T06:09:23.921214+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":274,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4152000,"kb_used_data":17728,"kb_used_omap":4,"kb_used_meta":4134203,"kb_avail":278963520,"statfs":{"total":289910292480,"available":285658644480,"internally_reserved":0,"allocated":18153472,"data_stored":10166470,"data_compressed":5974,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":4233424207},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":53,"apply_latency_ms":53,"commit_latency_ns":53000000,"apply_latency_ns":53000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536599724,"num_objects":-166,"num_object_clones":0,"num_object_copies":-332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-166,"num_whiteouts":0,"num_read":-140662,"num_read_kb":-39745087,"num_write":-600203,"num_write_kb":-217692715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002245"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1013,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-14T06:09:14.279288+0000","last_change":"2024-09-14T05:33:26.938952+0000","last_active":"2024-09-14T06:09:14.279288+0000","last_peered":"2024-09-14T06:09:14.279288+0000","last_clean":"2024-09-14T06:09:14.279288+0000","last_became_active":"2024-09-14T05:33:24.930256+0000","last_became_peered":"2024-09-14T05:33:24.930256+0000","last_unstale":"2024-09-14T06:09:14.279288+0000","last_undegraded":"2024-09-14T06:09:14.279288+0000","last_fullsized":"2024-09-14T06:09:14.279288+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_clean_scrub_stamp":"2024-09-14T05:33:23.917964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:50:40.298612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087908799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1013,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-14T06:09:14.280491+0000","last_change":"2024-09-14T05:33:26.939287+0000","last_active":"2024-09-14T06:09:14.280491+0000","last_peered":"2024-09-14T06:09:14.280491+0000","last_clean":"2024-09-14T06:09:14.280491+0000","last_became_active":"2024-09-14T05:33:24.932185+0000","last_became_peered":"2024-09-14T05:33:24.932185+0000","last_unstale":"2024-09-14T06:09:14.280491+0000","last_undegraded":"2024-09-14T06:09:14.280491+0000","last_fullsized":"2024-09-14T06:09:14.280491+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_clean_scrub_stamp":"2024-09-14T05:33:23.917964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:46:00.286124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001112699,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1014,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-14T06:09:14.280789+0000","last_change":"2024-09-14T05:33:26.938327+0000","last_active":"2024-09-14T06:09:14.280789+0000","last_peered":"2024-09-14T06:09:14.280789+0000","last_clean":"2024-09-14T06:09:14.280789+0000","last_became_active":"2024-09-14T05:33:24.932193+0000","last_became_peered":"2024-09-14T05:33:24.932193+0000","last_unstale":"2024-09-14T06:09:14.280789+0000","last_undegraded":"2024-09-14T06:09:14.280789+0000","last_fullsized":"2024-09-14T06:09:14.280789+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_clean_scrub_stamp":"2024-09-14T05:33:23.917964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:39:48.398667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059268600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1014,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-14T06:09:14.278876+0000","last_change":"2024-09-14T05:33:26.938386+0000","last_active":"2024-09-14T06:09:14.278876+0000","last_peered":"2024-09-14T06:09:14.278876+0000","last_clean":"2024-09-14T06:09:14.278876+0000","last_became_active":"2024-09-14T05:33:24.931776+0000","last_became_peered":"2024-09-14T05:33:24.931776+0000","last_unstale":"2024-09-14T06:09:14.278876+0000","last_undegraded":"2024-09-14T06:09:14.278876+0000","last_fullsized":"2024-09-14T06:09:14.278876+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_clean_scrub_stamp":"2024-09-14T05:33:23.917964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:47:43.623946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061101000000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":1017,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-14T06:09:20.293652+0000","last_change":"2024-09-14T05:33:26.935540+0000","last_active":"2024-09-14T06:09:20.293652+0000","last_peered":"2024-09-14T06:09:20.293652+0000","last_clean":"2024-09-14T06:09:20.293652+0000","last_became_active":"2024-09-14T05:33:24.931475+0000","last_became_peered":"2024-09-14T05:33:24.931475+0000","last_unstale":"2024-09-14T06:09:20.293652+0000","last_undegraded":"2024-09-14T06:09:20.293652+0000","last_fullsized":"2024-09-14T06:09:20.293652+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_clean_scrub_stamp":"2024-09-14T05:33:23.917964+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:22:40.175631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054839899999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1017,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-14T06:09:20.316150+0000","last_change":"2024-09-14T05:33:27.312188+0000","last_active":"2024-09-14T06:09:20.316150+0000","last_peered":"2024-09-14T06:09:20.316150+0000","last_clean":"2024-09-14T06:09:20.316150+0000","last_became_active":"2024-09-14T05:33:24.931144+0000","last_became_peered":"2024-09-14T05:33:24.931144+0000","last_unstale":"2024-09-14T06:09:20.316150+0000","last_undegraded":"2024-09-14T06:09:20.316150+0000","last_fullsized":"2024-09-14T06:09:20.316150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_clean_scrub_stamp":"2024-09-14T05:33:23.917964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:00:35.641527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037866499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1017,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-14T06:09:20.326595+0000","last_change":"2024-09-14T05:33:27.312182+0000","last_active":"2024-09-14T06:09:20.326595+0000","last_peered":"2024-09-14T06:09:20.326595+0000","last_clean":"2024-09-14T06:09:20.326595+0000","last_became_active":"2024-09-14T05:33:24.931485+0000","last_became_peered":"2024-09-14T05:33:24.931485+0000","last_unstale":"2024-09-14T06:09:20.326595+0000","last_undegraded":"2024-09-14T06:09:20.326595+0000","last_fullsized":"2024-09-14T06:09:20.326595+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_clean_scrub_stamp":"2024-09-14T05:33:23.917964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:40:15.748455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039837999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":1015,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-14T06:09:14.279247+0000","last_change":"2024-09-14T05:33:26.938911+0000","last_active":"2024-09-14T06:09:14.279247+0000","last_peered":"2024-09-14T06:09:14.279247+0000","last_clean":"2024-09-14T06:09:14.279247+0000","last_became_active":"2024-09-14T05:33:24.930770+0000","last_became_peered":"2024-09-14T05:33:24.930770+0000","last_unstale":"2024-09-14T06:09:14.279247+0000","last_undegraded":"2024-09-14T06:09:14.279247+0000","last_fullsized":"2024-09-14T06:09:14.279247+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_clean_scrub_stamp":"2024-09-14T05:33:23.917964+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:07:34.790624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010537719999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'52","reported_seq":1082,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-14T06:09:14.277460+0000","last_change":"2024-09-14T05:33:19.761735+0000","last_active":"2024-09-14T06:09:14.277460+0000","last_peered":"2024-09-14T06:09:14.277460+0000","last_clean":"2024-09-14T06:09:14.277460+0000","last_became_active":"2024-09-14T05:33:19.760730+0000","last_became_peered":"2024-09-14T05:33:19.760730+0000","last_unstale":"2024-09-14T06:09:14.277460+0000","last_undegraded":"2024-09-14T06:09:14.277460+0000","last_fullsized":"2024-09-14T06:09:14.277460+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:33:18.427520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:33:18.427520+0000","last_clean_scrub_stamp":"2024-09-14T05:33:18.427520+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:04:05.440382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5974,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654706101,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1323704,"kb_used_data":5920,"kb_used_omap":1,"kb_used_meta":1317758,"kb_avail":93048136,"statfs":{"total":96636764160,"available":95281291264,"internally_reserved":0,"allocated":6062080,"data_stored":3580864,"data_compressed":2987,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1349384645},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738805,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1229256,"kb_used_data":5876,"kb_used_omap":1,"kb_used_meta":1223358,"kb_avail":93142584,"statfs":{"total":96636764160,"available":95378006016,"internally_reserved":0,"allocated":6017024,"data_stored":2997619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1252719045},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738804,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1599040,"kb_used_data":5932,"kb_used_omap":1,"kb_used_meta":1593086,"kb_avail":92772800,"statfs":{"total":96636764160,"available":94999347200,"internally_reserved":0,"allocated":6074368,"data_stored":3587987,"data_compressed":2987,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1631320517},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2987,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2987,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T06:09:24.303 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-14T06:09:24.593 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T06:09:24.594 INFO:teuthology.orchestra.run.smithi191.stderr:dumped all 2024-09-14T06:09:24.606 INFO:teuthology.orchestra.run.smithi191.stdout:{"pg_ready":true,"pg_map":{"version":1589,"stamp":"2024-09-14T06:09:23.921214+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":274,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4152000,"kb_used_data":17728,"kb_used_omap":4,"kb_used_meta":4134203,"kb_avail":278963520,"statfs":{"total":289910292480,"available":285658644480,"internally_reserved":0,"allocated":18153472,"data_stored":10166470,"data_compressed":5974,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":4233424207},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":53,"apply_latency_ms":53,"commit_latency_ns":53000000,"apply_latency_ns":53000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536599724,"num_objects":-166,"num_object_clones":0,"num_object_copies":-332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-166,"num_whiteouts":0,"num_read":-140662,"num_read_kb":-39745087,"num_write":-600203,"num_write_kb":-217692715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002245"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1013,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-14T06:09:14.279288+0000","last_change":"2024-09-14T05:33:26.938952+0000","last_active":"2024-09-14T06:09:14.279288+0000","last_peered":"2024-09-14T06:09:14.279288+0000","last_clean":"2024-09-14T06:09:14.279288+0000","last_became_active":"2024-09-14T05:33:24.930256+0000","last_became_peered":"2024-09-14T05:33:24.930256+0000","last_unstale":"2024-09-14T06:09:14.279288+0000","last_undegraded":"2024-09-14T06:09:14.279288+0000","last_fullsized":"2024-09-14T06:09:14.279288+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_clean_scrub_stamp":"2024-09-14T05:33:23.917964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:50:40.298612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087908799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1013,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-14T06:09:14.280491+0000","last_change":"2024-09-14T05:33:26.939287+0000","last_active":"2024-09-14T06:09:14.280491+0000","last_peered":"2024-09-14T06:09:14.280491+0000","last_clean":"2024-09-14T06:09:14.280491+0000","last_became_active":"2024-09-14T05:33:24.932185+0000","last_became_peered":"2024-09-14T05:33:24.932185+0000","last_unstale":"2024-09-14T06:09:14.280491+0000","last_undegraded":"2024-09-14T06:09:14.280491+0000","last_fullsized":"2024-09-14T06:09:14.280491+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_clean_scrub_stamp":"2024-09-14T05:33:23.917964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:46:00.286124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001112699,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1014,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-14T06:09:14.280789+0000","last_change":"2024-09-14T05:33:26.938327+0000","last_active":"2024-09-14T06:09:14.280789+0000","last_peered":"2024-09-14T06:09:14.280789+0000","last_clean":"2024-09-14T06:09:14.280789+0000","last_became_active":"2024-09-14T05:33:24.932193+0000","last_became_peered":"2024-09-14T05:33:24.932193+0000","last_unstale":"2024-09-14T06:09:14.280789+0000","last_undegraded":"2024-09-14T06:09:14.280789+0000","last_fullsized":"2024-09-14T06:09:14.280789+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_clean_scrub_stamp":"2024-09-14T05:33:23.917964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:39:48.398667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059268600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1014,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-14T06:09:14.278876+0000","last_change":"2024-09-14T05:33:26.938386+0000","last_active":"2024-09-14T06:09:14.278876+0000","last_peered":"2024-09-14T06:09:14.278876+0000","last_clean":"2024-09-14T06:09:14.278876+0000","last_became_active":"2024-09-14T05:33:24.931776+0000","last_became_peered":"2024-09-14T05:33:24.931776+0000","last_unstale":"2024-09-14T06:09:14.278876+0000","last_undegraded":"2024-09-14T06:09:14.278876+0000","last_fullsized":"2024-09-14T06:09:14.278876+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_clean_scrub_stamp":"2024-09-14T05:33:23.917964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:47:43.623946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061101000000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":1017,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-14T06:09:20.293652+0000","last_change":"2024-09-14T05:33:26.935540+0000","last_active":"2024-09-14T06:09:20.293652+0000","last_peered":"2024-09-14T06:09:20.293652+0000","last_clean":"2024-09-14T06:09:20.293652+0000","last_became_active":"2024-09-14T05:33:24.931475+0000","last_became_peered":"2024-09-14T05:33:24.931475+0000","last_unstale":"2024-09-14T06:09:20.293652+0000","last_undegraded":"2024-09-14T06:09:20.293652+0000","last_fullsized":"2024-09-14T06:09:20.293652+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_clean_scrub_stamp":"2024-09-14T05:33:23.917964+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:22:40.175631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054839899999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1017,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-14T06:09:20.316150+0000","last_change":"2024-09-14T05:33:27.312188+0000","last_active":"2024-09-14T06:09:20.316150+0000","last_peered":"2024-09-14T06:09:20.316150+0000","last_clean":"2024-09-14T06:09:20.316150+0000","last_became_active":"2024-09-14T05:33:24.931144+0000","last_became_peered":"2024-09-14T05:33:24.931144+0000","last_unstale":"2024-09-14T06:09:20.316150+0000","last_undegraded":"2024-09-14T06:09:20.316150+0000","last_fullsized":"2024-09-14T06:09:20.316150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_clean_scrub_stamp":"2024-09-14T05:33:23.917964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:00:35.641527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037866499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1017,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-14T06:09:20.326595+0000","last_change":"2024-09-14T05:33:27.312182+0000","last_active":"2024-09-14T06:09:20.326595+0000","last_peered":"2024-09-14T06:09:20.326595+0000","last_clean":"2024-09-14T06:09:20.326595+0000","last_became_active":"2024-09-14T05:33:24.931485+0000","last_became_peered":"2024-09-14T05:33:24.931485+0000","last_unstale":"2024-09-14T06:09:20.326595+0000","last_undegraded":"2024-09-14T06:09:20.326595+0000","last_fullsized":"2024-09-14T06:09:20.326595+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_clean_scrub_stamp":"2024-09-14T05:33:23.917964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:40:15.748455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039837999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":1015,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-14T06:09:14.279247+0000","last_change":"2024-09-14T05:33:26.938911+0000","last_active":"2024-09-14T06:09:14.279247+0000","last_peered":"2024-09-14T06:09:14.279247+0000","last_clean":"2024-09-14T06:09:14.279247+0000","last_became_active":"2024-09-14T05:33:24.930770+0000","last_became_peered":"2024-09-14T05:33:24.930770+0000","last_unstale":"2024-09-14T06:09:14.279247+0000","last_undegraded":"2024-09-14T06:09:14.279247+0000","last_fullsized":"2024-09-14T06:09:14.279247+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_clean_scrub_stamp":"2024-09-14T05:33:23.917964+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:07:34.790624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010537719999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'52","reported_seq":1082,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-14T06:09:14.277460+0000","last_change":"2024-09-14T05:33:19.761735+0000","last_active":"2024-09-14T06:09:14.277460+0000","last_peered":"2024-09-14T06:09:14.277460+0000","last_clean":"2024-09-14T06:09:14.277460+0000","last_became_active":"2024-09-14T05:33:19.760730+0000","last_became_peered":"2024-09-14T05:33:19.760730+0000","last_unstale":"2024-09-14T06:09:14.277460+0000","last_undegraded":"2024-09-14T06:09:14.277460+0000","last_fullsized":"2024-09-14T06:09:14.277460+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:33:18.427520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:33:18.427520+0000","last_clean_scrub_stamp":"2024-09-14T05:33:18.427520+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:04:05.440382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5974,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654706101,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1323704,"kb_used_data":5920,"kb_used_omap":1,"kb_used_meta":1317758,"kb_avail":93048136,"statfs":{"total":96636764160,"available":95281291264,"internally_reserved":0,"allocated":6062080,"data_stored":3580864,"data_compressed":2987,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1349384645},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738805,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1229256,"kb_used_data":5876,"kb_used_omap":1,"kb_used_meta":1223358,"kb_avail":93142584,"statfs":{"total":96636764160,"available":95378006016,"internally_reserved":0,"allocated":6017024,"data_stored":2997619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1252719045},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738804,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1599040,"kb_used_data":5932,"kb_used_omap":1,"kb_used_meta":1593086,"kb_avail":92772800,"statfs":{"total":96636764160,"available":94999347200,"internally_reserved":0,"allocated":6074368,"data_stored":3587987,"data_compressed":2987,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1631320517},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2987,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2987,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T06:09:24.607 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-09-14T06:09:24.607 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-14T06:09:24.885 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T06:09:24.886 INFO:teuthology.orchestra.run.smithi191.stderr:dumped all 2024-09-14T06:09:24.897 INFO:teuthology.orchestra.run.smithi191.stdout:{"pg_ready":true,"pg_map":{"version":1589,"stamp":"2024-09-14T06:09:23.921214+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":274,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4152000,"kb_used_data":17728,"kb_used_omap":4,"kb_used_meta":4134203,"kb_avail":278963520,"statfs":{"total":289910292480,"available":285658644480,"internally_reserved":0,"allocated":18153472,"data_stored":10166470,"data_compressed":5974,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":4233424207},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":53,"apply_latency_ms":53,"commit_latency_ns":53000000,"apply_latency_ns":53000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536599724,"num_objects":-166,"num_object_clones":0,"num_object_copies":-332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-166,"num_whiteouts":0,"num_read":-140662,"num_read_kb":-39745087,"num_write":-600203,"num_write_kb":-217692715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002245"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1013,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-14T06:09:14.279288+0000","last_change":"2024-09-14T05:33:26.938952+0000","last_active":"2024-09-14T06:09:14.279288+0000","last_peered":"2024-09-14T06:09:14.279288+0000","last_clean":"2024-09-14T06:09:14.279288+0000","last_became_active":"2024-09-14T05:33:24.930256+0000","last_became_peered":"2024-09-14T05:33:24.930256+0000","last_unstale":"2024-09-14T06:09:14.279288+0000","last_undegraded":"2024-09-14T06:09:14.279288+0000","last_fullsized":"2024-09-14T06:09:14.279288+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_clean_scrub_stamp":"2024-09-14T05:33:23.917964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:50:40.298612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087908799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1013,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-14T06:09:14.280491+0000","last_change":"2024-09-14T05:33:26.939287+0000","last_active":"2024-09-14T06:09:14.280491+0000","last_peered":"2024-09-14T06:09:14.280491+0000","last_clean":"2024-09-14T06:09:14.280491+0000","last_became_active":"2024-09-14T05:33:24.932185+0000","last_became_peered":"2024-09-14T05:33:24.932185+0000","last_unstale":"2024-09-14T06:09:14.280491+0000","last_undegraded":"2024-09-14T06:09:14.280491+0000","last_fullsized":"2024-09-14T06:09:14.280491+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_clean_scrub_stamp":"2024-09-14T05:33:23.917964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:46:00.286124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001112699,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1014,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-14T06:09:14.280789+0000","last_change":"2024-09-14T05:33:26.938327+0000","last_active":"2024-09-14T06:09:14.280789+0000","last_peered":"2024-09-14T06:09:14.280789+0000","last_clean":"2024-09-14T06:09:14.280789+0000","last_became_active":"2024-09-14T05:33:24.932193+0000","last_became_peered":"2024-09-14T05:33:24.932193+0000","last_unstale":"2024-09-14T06:09:14.280789+0000","last_undegraded":"2024-09-14T06:09:14.280789+0000","last_fullsized":"2024-09-14T06:09:14.280789+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_clean_scrub_stamp":"2024-09-14T05:33:23.917964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:39:48.398667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059268600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1014,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-14T06:09:14.278876+0000","last_change":"2024-09-14T05:33:26.938386+0000","last_active":"2024-09-14T06:09:14.278876+0000","last_peered":"2024-09-14T06:09:14.278876+0000","last_clean":"2024-09-14T06:09:14.278876+0000","last_became_active":"2024-09-14T05:33:24.931776+0000","last_became_peered":"2024-09-14T05:33:24.931776+0000","last_unstale":"2024-09-14T06:09:14.278876+0000","last_undegraded":"2024-09-14T06:09:14.278876+0000","last_fullsized":"2024-09-14T06:09:14.278876+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_clean_scrub_stamp":"2024-09-14T05:33:23.917964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:47:43.623946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061101000000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":1017,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-14T06:09:20.293652+0000","last_change":"2024-09-14T05:33:26.935540+0000","last_active":"2024-09-14T06:09:20.293652+0000","last_peered":"2024-09-14T06:09:20.293652+0000","last_clean":"2024-09-14T06:09:20.293652+0000","last_became_active":"2024-09-14T05:33:24.931475+0000","last_became_peered":"2024-09-14T05:33:24.931475+0000","last_unstale":"2024-09-14T06:09:20.293652+0000","last_undegraded":"2024-09-14T06:09:20.293652+0000","last_fullsized":"2024-09-14T06:09:20.293652+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_clean_scrub_stamp":"2024-09-14T05:33:23.917964+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:22:40.175631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054839899999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1017,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-14T06:09:20.316150+0000","last_change":"2024-09-14T05:33:27.312188+0000","last_active":"2024-09-14T06:09:20.316150+0000","last_peered":"2024-09-14T06:09:20.316150+0000","last_clean":"2024-09-14T06:09:20.316150+0000","last_became_active":"2024-09-14T05:33:24.931144+0000","last_became_peered":"2024-09-14T05:33:24.931144+0000","last_unstale":"2024-09-14T06:09:20.316150+0000","last_undegraded":"2024-09-14T06:09:20.316150+0000","last_fullsized":"2024-09-14T06:09:20.316150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_clean_scrub_stamp":"2024-09-14T05:33:23.917964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:00:35.641527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037866499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1017,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-14T06:09:20.326595+0000","last_change":"2024-09-14T05:33:27.312182+0000","last_active":"2024-09-14T06:09:20.326595+0000","last_peered":"2024-09-14T06:09:20.326595+0000","last_clean":"2024-09-14T06:09:20.326595+0000","last_became_active":"2024-09-14T05:33:24.931485+0000","last_became_peered":"2024-09-14T05:33:24.931485+0000","last_unstale":"2024-09-14T06:09:20.326595+0000","last_undegraded":"2024-09-14T06:09:20.326595+0000","last_fullsized":"2024-09-14T06:09:20.326595+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_clean_scrub_stamp":"2024-09-14T05:33:23.917964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:40:15.748455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039837999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":1015,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-14T06:09:14.279247+0000","last_change":"2024-09-14T05:33:26.938911+0000","last_active":"2024-09-14T06:09:14.279247+0000","last_peered":"2024-09-14T06:09:14.279247+0000","last_clean":"2024-09-14T06:09:14.279247+0000","last_became_active":"2024-09-14T05:33:24.930770+0000","last_became_peered":"2024-09-14T05:33:24.930770+0000","last_unstale":"2024-09-14T06:09:14.279247+0000","last_undegraded":"2024-09-14T06:09:14.279247+0000","last_fullsized":"2024-09-14T06:09:14.279247+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_clean_scrub_stamp":"2024-09-14T05:33:23.917964+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:07:34.790624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010537719999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'52","reported_seq":1082,"reported_epoch":505,"state":"active+clean","last_fresh":"2024-09-14T06:09:14.277460+0000","last_change":"2024-09-14T05:33:19.761735+0000","last_active":"2024-09-14T06:09:14.277460+0000","last_peered":"2024-09-14T06:09:14.277460+0000","last_clean":"2024-09-14T06:09:14.277460+0000","last_became_active":"2024-09-14T05:33:19.760730+0000","last_became_peered":"2024-09-14T05:33:19.760730+0000","last_unstale":"2024-09-14T06:09:14.277460+0000","last_undegraded":"2024-09-14T06:09:14.277460+0000","last_fullsized":"2024-09-14T06:09:14.277460+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:33:18.427520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:33:18.427520+0000","last_clean_scrub_stamp":"2024-09-14T05:33:18.427520+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:04:05.440382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5974,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654706101,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1323704,"kb_used_data":5920,"kb_used_omap":1,"kb_used_meta":1317758,"kb_avail":93048136,"statfs":{"total":96636764160,"available":95281291264,"internally_reserved":0,"allocated":6062080,"data_stored":3580864,"data_compressed":2987,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1349384645},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738805,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1229256,"kb_used_data":5876,"kb_used_omap":1,"kb_used_meta":1223358,"kb_avail":93142584,"statfs":{"total":96636764160,"available":95378006016,"internally_reserved":0,"allocated":6017024,"data_stored":2997619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1252719045},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738804,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1599040,"kb_used_data":5932,"kb_used_omap":1,"kb_used_meta":1593086,"kb_avail":92772800,"statfs":{"total":96636764160,"available":94999347200,"internally_reserved":0,"allocated":6074368,"data_stored":3587987,"data_compressed":2987,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1631320517},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2987,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2987,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T06:09:24.899 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-14T06:09:25.185 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T06:09:25.185 INFO:teuthology.orchestra.run.smithi191.stdout:{"epoch":507,"fsid":"2fb493c9-6b9e-47da-a932-4884bef35060","created":"2024-09-14T05:33:07.928340+0000","modified":"2024-09-14T06:09:22.284741+0000","last_up_change":"2024-09-14T05:33:17.425068+0000","last_in_change":"2024-09-14T05:33:08.760360+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-14T05:33:17.570265+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-14T05:33:23.035711+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5275b373-5a00-4b1f-bbaf-b5a4d36f3d1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":1406167874},{"type":"v1","addr":"172.21.15.191:6801","nonce":1406167874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":1406167874},{"type":"v1","addr":"172.21.15.191:6803","nonce":1406167874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":1406167874},{"type":"v1","addr":"172.21.15.191:6807","nonce":1406167874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":1406167874},{"type":"v1","addr":"172.21.15.191:6805","nonce":1406167874}]},"public_addr":"172.21.15.191:6801/1406167874","cluster_addr":"172.21.15.191:6803/1406167874","heartbeat_back_addr":"172.21.15.191:6807/1406167874","heartbeat_front_addr":"172.21.15.191:6805/1406167874","state":["exists","up"]},{"osd":1,"uuid":"74412469-f84e-4221-9d46-e16391bc87c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":805361069},{"type":"v1","addr":"172.21.15.191:6817","nonce":805361069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":805361069},{"type":"v1","addr":"172.21.15.191:6819","nonce":805361069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":805361069},{"type":"v1","addr":"172.21.15.191:6823","nonce":805361069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":805361069},{"type":"v1","addr":"172.21.15.191:6821","nonce":805361069}]},"public_addr":"172.21.15.191:6817/805361069","cluster_addr":"172.21.15.191:6819/805361069","heartbeat_back_addr":"172.21.15.191:6823/805361069","heartbeat_front_addr":"172.21.15.191:6821/805361069","state":["exists","up"]},{"osd":2,"uuid":"015cffcb-af80-40d2-a86a-104cdb0167b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":913831784},{"type":"v1","addr":"172.21.15.191:6809","nonce":913831784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":913831784},{"type":"v1","addr":"172.21.15.191:6811","nonce":913831784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":913831784},{"type":"v1","addr":"172.21.15.191:6815","nonce":913831784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":913831784},{"type":"v1","addr":"172.21.15.191:6813","nonce":913831784}]},"public_addr":"172.21.15.191:6809/913831784","cluster_addr":"172.21.15.191:6811/913831784","heartbeat_back_addr":"172.21.15.191:6815/913831784","heartbeat_front_addr":"172.21.15.191:6813/913831784","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T05:33:15.060141+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T05:33:15.465487+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T05:33:15.286411+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.191:0/3302990693":"2024-09-14T07:08:34.009888+0000","172.21.15.191:0/160322713":"2024-09-14T07:06:25.341574+0000","172.21.15.191:0/2115390219":"2024-09-14T07:05:44.849122+0000","172.21.15.191:0/3193066106":"2024-09-14T07:03:42.707883+0000","172.21.15.191:0/2557496628":"2024-09-14T06:59:37.103581+0000","172.21.15.191:0/3045520399":"2024-09-14T06:55:28.993253+0000","172.21.15.191:0/1830733933":"2024-09-14T06:53:27.992487+0000","172.21.15.191:0/1882301153":"2024-09-14T06:54:55.130963+0000","172.21.15.191:0/1498428420":"2024-09-14T06:43:16.914935+0000","172.21.15.191:0/343710994":"2024-09-14T06:57:11.227687+0000","172.21.15.191:0/2562650500":"2024-09-14T06:54:13.016230+0000","172.21.15.191:0/1081589477":"2024-09-14T06:54:27.095500+0000","172.21.15.191:0/2984105596":"2024-09-14T06:55:37.035455+0000","172.21.15.191:0/304851029":"2024-09-14T06:54:05.944376+0000","172.21.15.191:0/1343698621":"2024-09-14T06:56:01.070687+0000","172.21.15.191:0/1008261876":"2024-09-14T06:53:58.977715+0000","172.21.15.191:0/186743253":"2024-09-14T07:09:12.394237+0000","172.21.15.191:0/2952532390":"2024-09-14T06:53:44.060995+0000","172.21.15.191:0/3793440824":"2024-09-14T06:54:34.077382+0000","172.21.15.191:0/906397190":"2024-09-14T07:04:21.403938+0000","172.21.15.191:0/3153100775":"2024-09-14T06:53:52.057489+0000","172.21.15.191:0/1594245296":"2024-09-14T06:58:48.087993+0000","172.21.15.191:0/20061648":"2024-09-14T06:54:41.106062+0000","172.21.15.191:0/4122378741":"2024-09-14T06:35:45.400840+0000","172.21.15.191:0/448912880":"2024-09-14T06:53:36.028875+0000","172.21.15.191:0/1289261805":"2024-09-14T06:36:32.519768+0000","172.21.15.191:0/3780814173":"2024-09-14T07:07:11.760501+0000","172.21.15.191:0/3502079819":"2024-09-14T06:37:19.044647+0000","172.21.15.191:0/1417140671":"2024-09-14T07:05:04.319007+0000","172.21.15.191:0/1034589099":"2024-09-14T06:56:15.005318+0000","172.21.15.191:0/428730430":"2024-09-14T06:39:02.921942+0000","172.21.15.191:0/3187253908":"2024-09-14T07:07:52.220230+0000","172.21.15.191:0/3105266475":"2024-09-14T06:41:37.513339+0000","172.21.15.191:0/504827260":"2024-09-14T06:55:53.060085+0000","172.21.15.191:0/3128691836":"2024-09-14T06:57:33.263056+0000","172.21.15.191:0/1047248267":"2024-09-14T06:38:07.831362+0000","172.21.15.191:0/2516267615":"2024-09-14T06:56:29.099489+0000","172.21.15.191:0/1942965770":"2024-09-14T06:42:24.090743+0000","172.21.15.191:0/944445211":"2024-09-14T06:58:14.202636+0000","172.21.15.191:0/3514880858":"2024-09-14T06:53:03.467275+0000","172.21.15.191:0/2892496040":"2024-09-14T06:56:43.135484+0000","172.21.15.191:0/946765581":"2024-09-14T07:03:03.810207+0000","172.21.15.191:0/3405176122":"2024-09-14T06:55:03.850815+0000","172.21.15.191:0/1507313941":"2024-09-14T06:58:31.023236+0000","172.21.15.191:0/953172088":"2024-09-14T06:54:20.051943+0000","172.21.15.191:0/3497529881":"2024-09-14T06:53:11.883798+0000","172.21.15.191:0/2977774002":"2024-09-14T06:57:04.212972+0000","172.21.15.191:0/2962209737":"2024-09-14T06:56:22.082446+0000","172.21.15.191:0/304596439":"2024-09-14T06:40:44.619019+0000","172.21.15.191:0/2584868414":"2024-09-14T06:53:19.992573+0000","172.21.15.191:0/3325079745":"2024-09-14T06:55:45.009101+0000","172.21.15.191:0/3214411005":"2024-09-14T06:56:08.025973+0000","172.21.15.191:0/3376263392":"2024-09-14T06:56:36.121402+0000","172.21.15.191:0/2022190846":"2024-09-14T06:56:57.170607+0000","172.21.15.191:0/1284233051":"2024-09-14T06:54:48.101952+0000","172.21.15.191:0/2110073895":"2024-09-14T06:57:18.225225+0000","172.21.15.191:0/3194316669":"2024-09-14T06:59:05.039684+0000","172.21.15.191:0/2463997855":"2024-09-14T06:57:26.915990+0000","172.21.15.191:0/1121503927":"2024-09-14T06:39:56.482478+0000","172.21.15.191:0/1744276959":"2024-09-14T06:57:49.978519+0000","172.21.15.191:0/239512226":"2024-09-14T06:57:57.984239+0000","172.21.15.191:0/1210612424":"2024-09-14T06:57:41.949960+0000","172.21.15.191:0/1139582474":"2024-09-14T06:58:23.024024+0000","172.21.15.191:0/725982910":"2024-09-14T06:58:39.286402+0000","172.21.15.191:0/4085960917":"2024-09-14T06:58:56.107553+0000","172.21.15.191:0/2008052399":"2024-09-14T06:56:50.176848+0000","172.21.15.191:0/629058483":"2024-09-14T06:59:13.056904+0000","172.21.15.191:0/1695497036":"2024-09-14T06:55:20.257344+0000","172.21.15.191:0/972851982":"2024-09-14T06:59:21.075689+0000","172.21.15.191:0/1001305109":"2024-09-14T06:59:29.067141+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":194,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T06:09:26.199 INFO:tasks.ceph:Scrubbing cluster2.osd.0 2024-09-14T06:09:26.199 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-14T06:09:26.335 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-14T06:09:26.335 INFO:teuthology.orchestra.run.smithi191.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-14T06:09:26.335 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-14T06:09:26.346 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 0 2024-09-14T06:09:26.630 INFO:teuthology.orchestra.run.smithi191.stderr:instructed osd(s) 0 to deep-scrub 2024-09-14T06:09:26.644 INFO:tasks.ceph:Scrubbing cluster2.osd.1 2024-09-14T06:09:26.644 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-14T06:09:26.765 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-14T06:09:26.765 INFO:teuthology.orchestra.run.smithi191.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-14T06:09:26.766 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-14T06:09:26.776 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 1 2024-09-14T06:09:27.061 INFO:teuthology.orchestra.run.smithi191.stderr:instructed osd(s) 1 to deep-scrub 2024-09-14T06:09:27.075 INFO:tasks.ceph:Scrubbing cluster2.osd.2 2024-09-14T06:09:27.075 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-14T06:09:27.198 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-14T06:09:27.199 INFO:teuthology.orchestra.run.smithi191.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-14T06:09:27.199 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-14T06:09:27.209 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 2 2024-09-14T06:09:27.493 INFO:teuthology.orchestra.run.smithi191.stderr:instructed osd(s) 2 to deep-scrub 2024-09-14T06:09:27.508 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-14T06:09:27.790 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T06:09:27.790 INFO:teuthology.orchestra.run.smithi191.stderr:dumped all 2024-09-14T06:09:27.803 INFO:teuthology.orchestra.run.smithi191.stdout:{"pg_ready":true,"pg_map":{"version":1590,"stamp":"2024-09-14T06:09:25.921479+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":185,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4170076,"kb_used_data":17756,"kb_used_omap":4,"kb_used_meta":4152251,"kb_avail":278945444,"statfs":{"total":289910292480,"available":285640134656,"internally_reserved":0,"allocated":18182144,"data_stored":10180261,"data_compressed":5974,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":4251905359},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":67,"apply_latency_ms":67,"commit_latency_ns":67000000,"apply_latency_ns":67000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-155070568,"num_objects":-73,"num_object_clones":0,"num_object_copies":-146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-73,"num_whiteouts":0,"num_read":-140695,"num_read_kb":-39745114,"num_write":-600306,"num_write_kb":-217692718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002011"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1017,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-14T06:09:22.450543+0000","last_change":"2024-09-14T05:33:26.938952+0000","last_active":"2024-09-14T06:09:22.450543+0000","last_peered":"2024-09-14T06:09:22.450543+0000","last_clean":"2024-09-14T06:09:22.450543+0000","last_became_active":"2024-09-14T05:33:24.930256+0000","last_became_peered":"2024-09-14T05:33:24.930256+0000","last_unstale":"2024-09-14T06:09:22.450543+0000","last_undegraded":"2024-09-14T06:09:22.450543+0000","last_fullsized":"2024-09-14T06:09:22.450543+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_clean_scrub_stamp":"2024-09-14T05:33:23.917964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:50:40.298612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087908799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1017,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-14T06:09:22.454801+0000","last_change":"2024-09-14T05:33:26.939287+0000","last_active":"2024-09-14T06:09:22.454801+0000","last_peered":"2024-09-14T06:09:22.454801+0000","last_clean":"2024-09-14T06:09:22.454801+0000","last_became_active":"2024-09-14T05:33:24.932185+0000","last_became_peered":"2024-09-14T05:33:24.932185+0000","last_unstale":"2024-09-14T06:09:22.454801+0000","last_undegraded":"2024-09-14T06:09:22.454801+0000","last_fullsized":"2024-09-14T06:09:22.454801+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_clean_scrub_stamp":"2024-09-14T05:33:23.917964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:46:00.286124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001112699,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1018,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-14T06:09:22.457296+0000","last_change":"2024-09-14T05:33:26.938327+0000","last_active":"2024-09-14T06:09:22.457296+0000","last_peered":"2024-09-14T06:09:22.457296+0000","last_clean":"2024-09-14T06:09:22.457296+0000","last_became_active":"2024-09-14T05:33:24.932193+0000","last_became_peered":"2024-09-14T05:33:24.932193+0000","last_unstale":"2024-09-14T06:09:22.457296+0000","last_undegraded":"2024-09-14T06:09:22.457296+0000","last_fullsized":"2024-09-14T06:09:22.457296+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_clean_scrub_stamp":"2024-09-14T05:33:23.917964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:39:48.398667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059268600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1018,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-14T06:09:22.452274+0000","last_change":"2024-09-14T05:33:26.938386+0000","last_active":"2024-09-14T06:09:22.452274+0000","last_peered":"2024-09-14T06:09:22.452274+0000","last_clean":"2024-09-14T06:09:22.452274+0000","last_became_active":"2024-09-14T05:33:24.931776+0000","last_became_peered":"2024-09-14T05:33:24.931776+0000","last_unstale":"2024-09-14T06:09:22.452274+0000","last_undegraded":"2024-09-14T06:09:22.452274+0000","last_fullsized":"2024-09-14T06:09:22.452274+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_clean_scrub_stamp":"2024-09-14T05:33:23.917964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:47:43.623946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061101000000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":1017,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-14T06:09:20.293652+0000","last_change":"2024-09-14T05:33:26.935540+0000","last_active":"2024-09-14T06:09:20.293652+0000","last_peered":"2024-09-14T06:09:20.293652+0000","last_clean":"2024-09-14T06:09:20.293652+0000","last_became_active":"2024-09-14T05:33:24.931475+0000","last_became_peered":"2024-09-14T05:33:24.931475+0000","last_unstale":"2024-09-14T06:09:20.293652+0000","last_undegraded":"2024-09-14T06:09:20.293652+0000","last_fullsized":"2024-09-14T06:09:20.293652+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_clean_scrub_stamp":"2024-09-14T05:33:23.917964+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:22:40.175631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054839899999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1017,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-14T06:09:20.316150+0000","last_change":"2024-09-14T05:33:27.312188+0000","last_active":"2024-09-14T06:09:20.316150+0000","last_peered":"2024-09-14T06:09:20.316150+0000","last_clean":"2024-09-14T06:09:20.316150+0000","last_became_active":"2024-09-14T05:33:24.931144+0000","last_became_peered":"2024-09-14T05:33:24.931144+0000","last_unstale":"2024-09-14T06:09:20.316150+0000","last_undegraded":"2024-09-14T06:09:20.316150+0000","last_fullsized":"2024-09-14T06:09:20.316150+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_clean_scrub_stamp":"2024-09-14T05:33:23.917964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:00:35.641527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037866499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1017,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-09-14T06:09:20.326595+0000","last_change":"2024-09-14T05:33:27.312182+0000","last_active":"2024-09-14T06:09:20.326595+0000","last_peered":"2024-09-14T06:09:20.326595+0000","last_clean":"2024-09-14T06:09:20.326595+0000","last_became_active":"2024-09-14T05:33:24.931485+0000","last_became_peered":"2024-09-14T05:33:24.931485+0000","last_unstale":"2024-09-14T06:09:20.326595+0000","last_undegraded":"2024-09-14T06:09:20.326595+0000","last_fullsized":"2024-09-14T06:09:20.326595+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_clean_scrub_stamp":"2024-09-14T05:33:23.917964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:40:15.748455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039837999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":1019,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-14T06:09:22.451599+0000","last_change":"2024-09-14T05:33:26.938911+0000","last_active":"2024-09-14T06:09:22.451599+0000","last_peered":"2024-09-14T06:09:22.451599+0000","last_clean":"2024-09-14T06:09:22.451599+0000","last_became_active":"2024-09-14T05:33:24.930770+0000","last_became_peered":"2024-09-14T05:33:24.930770+0000","last_unstale":"2024-09-14T06:09:22.451599+0000","last_undegraded":"2024-09-14T06:09:22.451599+0000","last_fullsized":"2024-09-14T06:09:22.451599+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:33:23.917964+0000","last_clean_scrub_stamp":"2024-09-14T05:33:23.917964+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:07:34.790624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010537719999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'52","reported_seq":1087,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-14T06:09:22.448360+0000","last_change":"2024-09-14T05:33:19.761735+0000","last_active":"2024-09-14T06:09:22.448360+0000","last_peered":"2024-09-14T06:09:22.448360+0000","last_clean":"2024-09-14T06:09:22.448360+0000","last_became_active":"2024-09-14T05:33:19.760730+0000","last_became_peered":"2024-09-14T05:33:19.760730+0000","last_unstale":"2024-09-14T06:09:22.448360+0000","last_undegraded":"2024-09-14T06:09:22.448360+0000","last_fullsized":"2024-09-14T06:09:22.448360+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:33:18.427520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:33:18.427520+0000","last_clean_scrub_stamp":"2024-09-14T05:33:18.427520+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:04:05.440382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5974,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654706102,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1341780,"kb_used_data":5948,"kb_used_omap":1,"kb_used_meta":1335806,"kb_avail":93030060,"statfs":{"total":96636764160,"available":95262781440,"internally_reserved":0,"allocated":6090752,"data_stored":3594655,"data_compressed":2987,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1367865797},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738805,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1229256,"kb_used_data":5876,"kb_used_omap":1,"kb_used_meta":1223358,"kb_avail":93142584,"statfs":{"total":96636764160,"available":95378006016,"internally_reserved":0,"allocated":6017024,"data_stored":2997619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1252719045},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738804,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1599040,"kb_used_data":5932,"kb_used_omap":1,"kb_used_meta":1593086,"kb_avail":92772800,"statfs":{"total":96636764160,"available":94999347200,"internally_reserved":0,"allocated":6074368,"data_stored":3587987,"data_compressed":2987,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1631320517},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2987,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2987,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T06:09:27.807 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-14T05:33:23.917964+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=5, tm_min=33, tm_sec=23, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=6, tm_min=9, tm_sec=25, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T06:09:27.807 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-14T05:33:23.917964+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=5, tm_min=33, tm_sec=23, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=6, tm_min=9, tm_sec=25, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T06:09:27.807 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-14T05:33:23.917964+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=5, tm_min=33, tm_sec=23, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=6, tm_min=9, tm_sec=25, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T06:09:27.807 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-14T05:33:23.917964+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=5, tm_min=33, tm_sec=23, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=6, tm_min=9, tm_sec=25, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T06:09:27.807 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-14T05:33:23.917964+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=5, tm_min=33, tm_sec=23, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=6, tm_min=9, tm_sec=25, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T06:09:27.807 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-14T05:33:23.917964+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=5, tm_min=33, tm_sec=23, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=6, tm_min=9, tm_sec=25, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T06:09:27.808 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-14T05:33:23.917964+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=5, tm_min=33, tm_sec=23, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=6, tm_min=9, tm_sec=25, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T06:09:27.808 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-14T05:33:23.917964+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=5, tm_min=33, tm_sec=23, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=6, tm_min=9, tm_sec=25, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T06:09:27.808 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-14T05:33:18.427520+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=5, tm_min=33, tm_sec=18, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=6, tm_min=9, tm_sec=25, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T06:09:27.808 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-14T06:09:47.809 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-14T06:09:48.105 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-14T06:09:48.106 INFO:teuthology.orchestra.run.smithi191.stderr:dumped all 2024-09-14T06:09:48.117 INFO:teuthology.orchestra.run.smithi191.stdout:{"pg_ready":true,"pg_map":{"version":1601,"stamp":"2024-09-14T06:09:47.925748+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4170088,"kb_used_data":17784,"kb_used_omap":4,"kb_used_meta":4152251,"kb_avail":278945432,"statfs":{"total":289910292480,"available":285640122368,"internally_reserved":0,"allocated":18210816,"data_stored":10193578,"data_compressed":5974,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":4251905359},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002427"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1025,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-14T06:09:30.399485+0000","last_change":"2024-09-14T06:09:30.399485+0000","last_active":"2024-09-14T06:09:30.399485+0000","last_peered":"2024-09-14T06:09:30.399485+0000","last_clean":"2024-09-14T06:09:30.399485+0000","last_became_active":"2024-09-14T05:33:24.930256+0000","last_became_peered":"2024-09-14T05:33:24.930256+0000","last_unstale":"2024-09-14T06:09:30.399485+0000","last_undegraded":"2024-09-14T06:09:30.399485+0000","last_fullsized":"2024-09-14T06:09:30.399485+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:09:30.399391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:09:30.399391+0000","last_clean_scrub_stamp":"2024-09-14T06:09:30.399391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T06:41:14.901051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087908799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1025,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-14T06:09:31.386527+0000","last_change":"2024-09-14T06:09:31.386527+0000","last_active":"2024-09-14T06:09:31.386527+0000","last_peered":"2024-09-14T06:09:31.386527+0000","last_clean":"2024-09-14T06:09:31.386527+0000","last_became_active":"2024-09-14T05:33:24.932185+0000","last_became_peered":"2024-09-14T05:33:24.932185+0000","last_unstale":"2024-09-14T06:09:31.386527+0000","last_undegraded":"2024-09-14T06:09:31.386527+0000","last_fullsized":"2024-09-14T06:09:31.386527+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:09:31.386424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:09:31.386424+0000","last_clean_scrub_stamp":"2024-09-14T06:09:31.386424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:16:55.814561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001112699,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1026,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-14T06:09:32.432462+0000","last_change":"2024-09-14T06:09:32.432462+0000","last_active":"2024-09-14T06:09:32.432462+0000","last_peered":"2024-09-14T06:09:32.432462+0000","last_clean":"2024-09-14T06:09:32.432462+0000","last_became_active":"2024-09-14T05:33:24.932193+0000","last_became_peered":"2024-09-14T05:33:24.932193+0000","last_unstale":"2024-09-14T06:09:32.432462+0000","last_undegraded":"2024-09-14T06:09:32.432462+0000","last_fullsized":"2024-09-14T06:09:32.432462+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:09:32.432375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:09:32.432375+0000","last_clean_scrub_stamp":"2024-09-14T06:09:32.432375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:22:27.536082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059268600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1026,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-14T06:09:29.430229+0000","last_change":"2024-09-14T06:09:29.430229+0000","last_active":"2024-09-14T06:09:29.430229+0000","last_peered":"2024-09-14T06:09:29.430229+0000","last_clean":"2024-09-14T06:09:29.430229+0000","last_became_active":"2024-09-14T05:33:24.931776+0000","last_became_peered":"2024-09-14T05:33:24.931776+0000","last_unstale":"2024-09-14T06:09:29.430229+0000","last_undegraded":"2024-09-14T06:09:29.430229+0000","last_fullsized":"2024-09-14T06:09:29.430229+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:09:29.430144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:09:29.430144+0000","last_clean_scrub_stamp":"2024-09-14T06:09:29.430144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T17:20:25.451180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061101000000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"507'3","reported_seq":1028,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-14T06:09:26.886309+0000","last_change":"2024-09-14T06:09:26.886309+0000","last_active":"2024-09-14T06:09:26.886309+0000","last_peered":"2024-09-14T06:09:26.886309+0000","last_clean":"2024-09-14T06:09:26.886309+0000","last_became_active":"2024-09-14T05:33:24.931475+0000","last_became_peered":"2024-09-14T05:33:24.931475+0000","last_unstale":"2024-09-14T06:09:26.886309+0000","last_undegraded":"2024-09-14T06:09:26.886309+0000","last_fullsized":"2024-09-14T06:09:26.886309+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"507'3","last_scrub_stamp":"2024-09-14T06:09:26.886215+0000","last_deep_scrub":"507'3","last_deep_scrub_stamp":"2024-09-14T06:09:26.886215+0000","last_clean_scrub_stamp":"2024-09-14T06:09:26.886215+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:58:45.205408+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00054839899999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1027,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-14T06:09:29.315473+0000","last_change":"2024-09-14T06:09:29.315473+0000","last_active":"2024-09-14T06:09:29.315473+0000","last_peered":"2024-09-14T06:09:29.315473+0000","last_clean":"2024-09-14T06:09:29.315473+0000","last_became_active":"2024-09-14T05:33:24.931144+0000","last_became_peered":"2024-09-14T05:33:24.931144+0000","last_unstale":"2024-09-14T06:09:29.315473+0000","last_undegraded":"2024-09-14T06:09:29.315473+0000","last_fullsized":"2024-09-14T06:09:29.315473+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:09:29.315382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:09:29.315382+0000","last_clean_scrub_stamp":"2024-09-14T06:09:29.315382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:53:23.135884+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00037866499999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1028,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-14T06:09:28.327160+0000","last_change":"2024-09-14T06:09:28.327111+0000","last_active":"2024-09-14T06:09:28.327160+0000","last_peered":"2024-09-14T06:09:28.327160+0000","last_clean":"2024-09-14T06:09:28.327160+0000","last_became_active":"2024-09-14T05:33:24.931485+0000","last_became_peered":"2024-09-14T05:33:24.931485+0000","last_unstale":"2024-09-14T06:09:28.327160+0000","last_undegraded":"2024-09-14T06:09:28.327160+0000","last_fullsized":"2024-09-14T06:09:28.327160+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:09:28.327035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:09:28.327035+0000","last_clean_scrub_stamp":"2024-09-14T06:09:28.327035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:28:12.361274+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00039837999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"507'2","reported_seq":1028,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-14T06:09:27.430225+0000","last_change":"2024-09-14T06:09:27.430225+0000","last_active":"2024-09-14T06:09:27.430225+0000","last_peered":"2024-09-14T06:09:27.430225+0000","last_clean":"2024-09-14T06:09:27.430225+0000","last_became_active":"2024-09-14T05:33:24.930770+0000","last_became_peered":"2024-09-14T05:33:24.930770+0000","last_unstale":"2024-09-14T06:09:27.430225+0000","last_undegraded":"2024-09-14T06:09:27.430225+0000","last_fullsized":"2024-09-14T06:09:27.430225+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"507'2","last_scrub_stamp":"2024-09-14T06:09:27.430153+0000","last_deep_scrub":"507'2","last_deep_scrub_stamp":"2024-09-14T06:09:27.430153+0000","last_clean_scrub_stamp":"2024-09-14T06:09:27.430153+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T15:24:13.042202+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0010537719999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"507'54","reported_seq":1097,"reported_epoch":507,"state":"active+clean","last_fresh":"2024-09-14T06:09:28.388825+0000","last_change":"2024-09-14T06:09:28.388825+0000","last_active":"2024-09-14T06:09:28.388825+0000","last_peered":"2024-09-14T06:09:28.388825+0000","last_clean":"2024-09-14T06:09:28.388825+0000","last_became_active":"2024-09-14T05:33:19.760730+0000","last_became_peered":"2024-09-14T05:33:19.760730+0000","last_unstale":"2024-09-14T06:09:28.388825+0000","last_undegraded":"2024-09-14T06:09:28.388825+0000","last_fullsized":"2024-09-14T06:09:28.388825+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"507'54","last_scrub_stamp":"2024-09-14T06:09:28.388748+0000","last_deep_scrub":"507'54","last_deep_scrub_stamp":"2024-09-14T06:09:28.388748+0000","last_clean_scrub_stamp":"2024-09-14T06:09:28.388748+0000","objects_scrubbed":2,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:07:26.417417+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5974,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654706106,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1341772,"kb_used_data":5948,"kb_used_omap":1,"kb_used_meta":1335806,"kb_avail":93030068,"statfs":{"total":96636764160,"available":95262789632,"internally_reserved":0,"allocated":6090752,"data_stored":3594655,"data_compressed":2987,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1367865797},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738810,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1229264,"kb_used_data":5888,"kb_used_omap":1,"kb_used_meta":1223358,"kb_avail":93142576,"statfs":{"total":96636764160,"available":95377997824,"internally_reserved":0,"allocated":6029312,"data_stored":3004268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1252719045},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1599052,"kb_used_data":5948,"kb_used_omap":1,"kb_used_meta":1593086,"kb_avail":92772788,"statfs":{"total":96636764160,"available":94999334912,"internally_reserved":0,"allocated":6090752,"data_stored":3594655,"data_compressed":2987,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1631320517},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2987,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2987,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T06:09:48.119 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster cluster2 config set global mon_health_to_clog false 2024-09-14T06:09:48.419 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-14T06:09:48.419 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-14T06:09:48.419 DEBUG:tasks.ceph.cluster2.osd.0:waiting for process to exit 2024-09-14T06:09:48.419 INFO:teuthology.orchestra.run:waiting for 300 2024-09-14T06:09:48.495 INFO:tasks.ceph.cluster2.osd.0:Stopped 2024-09-14T06:09:48.495 DEBUG:tasks.ceph.cluster2.osd.1:waiting for process to exit 2024-09-14T06:09:48.495 INFO:teuthology.orchestra.run:waiting for 300 2024-09-14T06:09:48.576 INFO:tasks.ceph.cluster2.osd.1:Stopped 2024-09-14T06:09:48.577 DEBUG:tasks.ceph.cluster2.osd.2:waiting for process to exit 2024-09-14T06:09:48.577 INFO:teuthology.orchestra.run:waiting for 300 2024-09-14T06:09:48.646 INFO:tasks.ceph.cluster2.osd.2:Stopped 2024-09-14T06:09:48.646 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-14T06:09:48.647 DEBUG:tasks.ceph.cluster2.mgr.x:waiting for process to exit 2024-09-14T06:09:48.647 INFO:teuthology.orchestra.run:waiting for 300 2024-09-14T06:09:48.678 INFO:tasks.ceph.cluster2.mgr.x:Stopped 2024-09-14T06:09:48.678 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-14T06:09:48.678 DEBUG:tasks.ceph.cluster2.mon.a:waiting for process to exit 2024-09-14T06:09:48.679 INFO:teuthology.orchestra.run:waiting for 300 2024-09-14T06:09:48.691 INFO:tasks.ceph.cluster2.mon.a:Stopped 2024-09-14T06:09:48.692 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-14T06:09:48.692 DEBUG:teuthology.orchestra.run.smithi191:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-14T06:09:48.793 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-0 on ubuntu@smithi191.front.sepia.ceph.com 2024-09-14T06:09:48.793 DEBUG:teuthology.orchestra.run.smithi191:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-0 2024-09-14T06:09:49.102 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-1 on ubuntu@smithi191.front.sepia.ceph.com 2024-09-14T06:09:49.103 DEBUG:teuthology.orchestra.run.smithi191:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-1 2024-09-14T06:09:49.239 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-2 on ubuntu@smithi191.front.sepia.ceph.com 2024-09-14T06:09:49.240 DEBUG:teuthology.orchestra.run.smithi191:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-2 2024-09-14T06:09:49.358 INFO:tasks.ceph:Archiving mon data... 2024-09-14T06:09:49.361 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/lib/ceph/mon/cluster2-a to /home/teuthworker/archive/teuthology-2024-09-10_20:16:17-rbd-main-distro-default-smithi/7900324/data/cluster2.mon.a.tgz 2024-09-14T06:09:49.361 DEBUG:teuthology.orchestra.run.smithi191:> mktemp 2024-09-14T06:09:49.375 INFO:teuthology.orchestra.run.smithi191.stdout:/tmp/tmp.R44qTdBm08 2024-09-14T06:09:49.376 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster2-a -- . > /tmp/tmp.R44qTdBm08 2024-09-14T06:09:49.546 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 0666 /tmp/tmp.R44qTdBm08 2024-09-14T06:09:49.657 DEBUG:teuthology.orchestra.remote:smithi191:/tmp/tmp.R44qTdBm08 is 478KB 2024-09-14T06:09:49.728 DEBUG:teuthology.orchestra.run.smithi191:> rm -fr /tmp/tmp.R44qTdBm08 2024-09-14T06:09:49.745 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-14T06:09:49.745 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-09-14T06:09:49.748 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-09-14T06:09:49.821 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-14T06:09:49.833 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-09-14T06:09:49.833 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-14T06:09:50.127 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T06:09:50.127 INFO:teuthology.orchestra.run.smithi143.stderr:dumped all 2024-09-14T06:09:50.139 INFO:teuthology.orchestra.run.smithi143.stdout:{"pg_ready":true,"pg_map":{"version":1594,"stamp":"2024-09-14T06:09:49.777135+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":456364,"kb_used_data":11688,"kb_used_omap":4,"kb_used_meta":444603,"kb_avail":282659156,"statfs":{"total":289910292480,"available":289442975744,"internally_reserved":0,"allocated":11968512,"data_stored":7507879,"data_compressed":5974,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4783,"internal_metadata":455273809},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002571"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":963,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-14T06:09:22.248150+0000","last_change":"2024-09-14T05:32:47.301032+0000","last_active":"2024-09-14T06:09:22.248150+0000","last_peered":"2024-09-14T06:09:22.248150+0000","last_clean":"2024-09-14T06:09:22.248150+0000","last_became_active":"2024-09-14T05:32:45.297981+0000","last_became_peered":"2024-09-14T05:32:45.297981+0000","last_unstale":"2024-09-14T06:09:22.248150+0000","last_undegraded":"2024-09-14T06:09:22.248150+0000","last_fullsized":"2024-09-14T06:09:22.248150+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_clean_scrub_stamp":"2024-09-14T05:32:44.283339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:13:28.873678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059651699999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":963,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-14T06:09:22.246531+0000","last_change":"2024-09-14T05:32:47.301315+0000","last_active":"2024-09-14T06:09:22.246531+0000","last_peered":"2024-09-14T06:09:22.246531+0000","last_clean":"2024-09-14T06:09:22.246531+0000","last_became_active":"2024-09-14T05:32:45.296192+0000","last_became_peered":"2024-09-14T05:32:45.296192+0000","last_unstale":"2024-09-14T06:09:22.246531+0000","last_undegraded":"2024-09-14T06:09:22.246531+0000","last_fullsized":"2024-09-14T06:09:22.246531+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_clean_scrub_stamp":"2024-09-14T05:32:44.283339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:48:19.029283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079042800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":963,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-14T06:09:22.247934+0000","last_change":"2024-09-14T05:32:47.301413+0000","last_active":"2024-09-14T06:09:22.247934+0000","last_peered":"2024-09-14T06:09:22.247934+0000","last_clean":"2024-09-14T06:09:22.247934+0000","last_became_active":"2024-09-14T05:32:45.296187+0000","last_became_peered":"2024-09-14T05:32:45.296187+0000","last_unstale":"2024-09-14T06:09:22.247934+0000","last_undegraded":"2024-09-14T06:09:22.247934+0000","last_fullsized":"2024-09-14T06:09:22.247934+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_clean_scrub_stamp":"2024-09-14T05:32:44.283339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:07:34.288707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089357300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":963,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-14T06:09:22.245031+0000","last_change":"2024-09-14T05:32:47.301642+0000","last_active":"2024-09-14T06:09:22.245031+0000","last_peered":"2024-09-14T06:09:22.245031+0000","last_clean":"2024-09-14T06:09:22.245031+0000","last_became_active":"2024-09-14T05:32:45.296113+0000","last_became_peered":"2024-09-14T05:32:45.296113+0000","last_unstale":"2024-09-14T06:09:22.245031+0000","last_undegraded":"2024-09-14T06:09:22.245031+0000","last_fullsized":"2024-09-14T06:09:22.245031+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_clean_scrub_stamp":"2024-09-14T05:32:44.283339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:11:57.020807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088137700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":965,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-14T06:09:21.966886+0000","last_change":"2024-09-14T05:32:47.298252+0000","last_active":"2024-09-14T06:09:21.966886+0000","last_peered":"2024-09-14T06:09:21.966886+0000","last_clean":"2024-09-14T06:09:21.966886+0000","last_became_active":"2024-09-14T05:32:45.298707+0000","last_became_peered":"2024-09-14T05:32:45.298707+0000","last_unstale":"2024-09-14T06:09:21.966886+0000","last_undegraded":"2024-09-14T06:09:21.966886+0000","last_fullsized":"2024-09-14T06:09:21.966886+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_clean_scrub_stamp":"2024-09-14T05:32:44.283339+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:20:40.111046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054093499999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":961,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-14T06:09:21.113522+0000","last_change":"2024-09-14T05:32:47.511782+0000","last_active":"2024-09-14T06:09:21.113522+0000","last_peered":"2024-09-14T06:09:21.113522+0000","last_clean":"2024-09-14T06:09:21.113522+0000","last_became_active":"2024-09-14T05:32:45.296961+0000","last_became_peered":"2024-09-14T05:32:45.296961+0000","last_unstale":"2024-09-14T06:09:21.113522+0000","last_undegraded":"2024-09-14T06:09:21.113522+0000","last_fullsized":"2024-09-14T06:09:21.113522+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_clean_scrub_stamp":"2024-09-14T05:32:44.283339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:49:29.999927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031355,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":962,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-14T06:09:21.114505+0000","last_change":"2024-09-14T05:32:47.511875+0000","last_active":"2024-09-14T06:09:21.114505+0000","last_peered":"2024-09-14T06:09:21.114505+0000","last_clean":"2024-09-14T06:09:21.114505+0000","last_became_active":"2024-09-14T05:32:45.298668+0000","last_became_peered":"2024-09-14T05:32:45.298668+0000","last_unstale":"2024-09-14T06:09:21.114505+0000","last_undegraded":"2024-09-14T06:09:21.114505+0000","last_fullsized":"2024-09-14T06:09:21.114505+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_clean_scrub_stamp":"2024-09-14T05:32:44.283339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T16:47:21.555137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035534800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":964,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-14T06:09:22.248801+0000","last_change":"2024-09-14T05:32:47.301008+0000","last_active":"2024-09-14T06:09:22.248801+0000","last_peered":"2024-09-14T06:09:22.248801+0000","last_clean":"2024-09-14T06:09:22.248801+0000","last_became_active":"2024-09-14T05:32:45.295172+0000","last_became_peered":"2024-09-14T05:32:45.295172+0000","last_unstale":"2024-09-14T06:09:22.248801+0000","last_undegraded":"2024-09-14T06:09:22.248801+0000","last_fullsized":"2024-09-14T06:09:22.248801+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_clean_scrub_stamp":"2024-09-14T05:32:44.283339+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:38:02.333228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061610699999999996,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":1031,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-14T06:09:22.247686+0000","last_change":"2024-09-14T05:32:41.276119+0000","last_active":"2024-09-14T06:09:22.247686+0000","last_peered":"2024-09-14T06:09:22.247686+0000","last_clean":"2024-09-14T06:09:22.247686+0000","last_became_active":"2024-09-14T05:32:41.275140+0000","last_became_peered":"2024-09-14T05:32:41.275140+0000","last_unstale":"2024-09-14T06:09:22.247686+0000","last_undegraded":"2024-09-14T06:09:22.247686+0000","last_fullsized":"2024-09-14T06:09:22.247686+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:32:40.262579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:32:40.262579+0000","last_clean_scrub_stamp":"2024-09-14T05:32:40.262579+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:45:44.677928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5974,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654706114,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":129504,"kb_used_data":3916,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":94242336,"statfs":{"total":96636764160,"available":96504152064,"internally_reserved":0,"allocated":4009984,"data_stored":2699422,"data_compressed":2987,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706113,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":165540,"kb_used_data":3916,"kb_used_omap":1,"kb_used_meta":161598,"kb_avail":94206300,"statfs":{"total":96636764160,"available":96467251200,"internally_reserved":0,"allocated":4009984,"data_stored":2699422,"data_compressed":2987,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":165476807},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738819,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":161320,"kb_used_data":3856,"kb_used_omap":1,"kb_used_meta":157438,"kb_avail":94210520,"statfs":{"total":96636764160,"available":96471572480,"internally_reserved":0,"allocated":3948544,"data_stored":2109035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":161216965},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2987,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2987,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T06:09:50.140 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-14T06:09:50.420 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T06:09:50.421 INFO:teuthology.orchestra.run.smithi143.stderr:dumped all 2024-09-14T06:09:50.433 INFO:teuthology.orchestra.run.smithi143.stdout:{"pg_ready":true,"pg_map":{"version":1594,"stamp":"2024-09-14T06:09:49.777135+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":456364,"kb_used_data":11688,"kb_used_omap":4,"kb_used_meta":444603,"kb_avail":282659156,"statfs":{"total":289910292480,"available":289442975744,"internally_reserved":0,"allocated":11968512,"data_stored":7507879,"data_compressed":5974,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4783,"internal_metadata":455273809},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002571"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":963,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-14T06:09:22.248150+0000","last_change":"2024-09-14T05:32:47.301032+0000","last_active":"2024-09-14T06:09:22.248150+0000","last_peered":"2024-09-14T06:09:22.248150+0000","last_clean":"2024-09-14T06:09:22.248150+0000","last_became_active":"2024-09-14T05:32:45.297981+0000","last_became_peered":"2024-09-14T05:32:45.297981+0000","last_unstale":"2024-09-14T06:09:22.248150+0000","last_undegraded":"2024-09-14T06:09:22.248150+0000","last_fullsized":"2024-09-14T06:09:22.248150+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_clean_scrub_stamp":"2024-09-14T05:32:44.283339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:13:28.873678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059651699999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":963,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-14T06:09:22.246531+0000","last_change":"2024-09-14T05:32:47.301315+0000","last_active":"2024-09-14T06:09:22.246531+0000","last_peered":"2024-09-14T06:09:22.246531+0000","last_clean":"2024-09-14T06:09:22.246531+0000","last_became_active":"2024-09-14T05:32:45.296192+0000","last_became_peered":"2024-09-14T05:32:45.296192+0000","last_unstale":"2024-09-14T06:09:22.246531+0000","last_undegraded":"2024-09-14T06:09:22.246531+0000","last_fullsized":"2024-09-14T06:09:22.246531+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_clean_scrub_stamp":"2024-09-14T05:32:44.283339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:48:19.029283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079042800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":963,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-14T06:09:22.247934+0000","last_change":"2024-09-14T05:32:47.301413+0000","last_active":"2024-09-14T06:09:22.247934+0000","last_peered":"2024-09-14T06:09:22.247934+0000","last_clean":"2024-09-14T06:09:22.247934+0000","last_became_active":"2024-09-14T05:32:45.296187+0000","last_became_peered":"2024-09-14T05:32:45.296187+0000","last_unstale":"2024-09-14T06:09:22.247934+0000","last_undegraded":"2024-09-14T06:09:22.247934+0000","last_fullsized":"2024-09-14T06:09:22.247934+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_clean_scrub_stamp":"2024-09-14T05:32:44.283339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:07:34.288707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089357300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":963,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-14T06:09:22.245031+0000","last_change":"2024-09-14T05:32:47.301642+0000","last_active":"2024-09-14T06:09:22.245031+0000","last_peered":"2024-09-14T06:09:22.245031+0000","last_clean":"2024-09-14T06:09:22.245031+0000","last_became_active":"2024-09-14T05:32:45.296113+0000","last_became_peered":"2024-09-14T05:32:45.296113+0000","last_unstale":"2024-09-14T06:09:22.245031+0000","last_undegraded":"2024-09-14T06:09:22.245031+0000","last_fullsized":"2024-09-14T06:09:22.245031+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_clean_scrub_stamp":"2024-09-14T05:32:44.283339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:11:57.020807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088137700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":965,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-14T06:09:21.966886+0000","last_change":"2024-09-14T05:32:47.298252+0000","last_active":"2024-09-14T06:09:21.966886+0000","last_peered":"2024-09-14T06:09:21.966886+0000","last_clean":"2024-09-14T06:09:21.966886+0000","last_became_active":"2024-09-14T05:32:45.298707+0000","last_became_peered":"2024-09-14T05:32:45.298707+0000","last_unstale":"2024-09-14T06:09:21.966886+0000","last_undegraded":"2024-09-14T06:09:21.966886+0000","last_fullsized":"2024-09-14T06:09:21.966886+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_clean_scrub_stamp":"2024-09-14T05:32:44.283339+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:20:40.111046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054093499999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":961,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-14T06:09:21.113522+0000","last_change":"2024-09-14T05:32:47.511782+0000","last_active":"2024-09-14T06:09:21.113522+0000","last_peered":"2024-09-14T06:09:21.113522+0000","last_clean":"2024-09-14T06:09:21.113522+0000","last_became_active":"2024-09-14T05:32:45.296961+0000","last_became_peered":"2024-09-14T05:32:45.296961+0000","last_unstale":"2024-09-14T06:09:21.113522+0000","last_undegraded":"2024-09-14T06:09:21.113522+0000","last_fullsized":"2024-09-14T06:09:21.113522+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_clean_scrub_stamp":"2024-09-14T05:32:44.283339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:49:29.999927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031355,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":962,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-14T06:09:21.114505+0000","last_change":"2024-09-14T05:32:47.511875+0000","last_active":"2024-09-14T06:09:21.114505+0000","last_peered":"2024-09-14T06:09:21.114505+0000","last_clean":"2024-09-14T06:09:21.114505+0000","last_became_active":"2024-09-14T05:32:45.298668+0000","last_became_peered":"2024-09-14T05:32:45.298668+0000","last_unstale":"2024-09-14T06:09:21.114505+0000","last_undegraded":"2024-09-14T06:09:21.114505+0000","last_fullsized":"2024-09-14T06:09:21.114505+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_clean_scrub_stamp":"2024-09-14T05:32:44.283339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T16:47:21.555137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035534800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":964,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-14T06:09:22.248801+0000","last_change":"2024-09-14T05:32:47.301008+0000","last_active":"2024-09-14T06:09:22.248801+0000","last_peered":"2024-09-14T06:09:22.248801+0000","last_clean":"2024-09-14T06:09:22.248801+0000","last_became_active":"2024-09-14T05:32:45.295172+0000","last_became_peered":"2024-09-14T05:32:45.295172+0000","last_unstale":"2024-09-14T06:09:22.248801+0000","last_undegraded":"2024-09-14T06:09:22.248801+0000","last_fullsized":"2024-09-14T06:09:22.248801+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_clean_scrub_stamp":"2024-09-14T05:32:44.283339+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:38:02.333228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061610699999999996,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":1031,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-14T06:09:22.247686+0000","last_change":"2024-09-14T05:32:41.276119+0000","last_active":"2024-09-14T06:09:22.247686+0000","last_peered":"2024-09-14T06:09:22.247686+0000","last_clean":"2024-09-14T06:09:22.247686+0000","last_became_active":"2024-09-14T05:32:41.275140+0000","last_became_peered":"2024-09-14T05:32:41.275140+0000","last_unstale":"2024-09-14T06:09:22.247686+0000","last_undegraded":"2024-09-14T06:09:22.247686+0000","last_fullsized":"2024-09-14T06:09:22.247686+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:32:40.262579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:32:40.262579+0000","last_clean_scrub_stamp":"2024-09-14T05:32:40.262579+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:45:44.677928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5974,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654706114,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":129504,"kb_used_data":3916,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":94242336,"statfs":{"total":96636764160,"available":96504152064,"internally_reserved":0,"allocated":4009984,"data_stored":2699422,"data_compressed":2987,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706113,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":165540,"kb_used_data":3916,"kb_used_omap":1,"kb_used_meta":161598,"kb_avail":94206300,"statfs":{"total":96636764160,"available":96467251200,"internally_reserved":0,"allocated":4009984,"data_stored":2699422,"data_compressed":2987,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":165476807},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738819,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":161320,"kb_used_data":3856,"kb_used_omap":1,"kb_used_meta":157438,"kb_avail":94210520,"statfs":{"total":96636764160,"available":96471572480,"internally_reserved":0,"allocated":3948544,"data_stored":2109035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":161216965},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2987,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2987,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T06:09:50.434 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-09-14T06:09:50.434 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-14T06:09:50.711 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T06:09:50.711 INFO:teuthology.orchestra.run.smithi143.stderr:dumped all 2024-09-14T06:09:50.723 INFO:teuthology.orchestra.run.smithi143.stdout:{"pg_ready":true,"pg_map":{"version":1594,"stamp":"2024-09-14T06:09:49.777135+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":456364,"kb_used_data":11688,"kb_used_omap":4,"kb_used_meta":444603,"kb_avail":282659156,"statfs":{"total":289910292480,"available":289442975744,"internally_reserved":0,"allocated":11968512,"data_stored":7507879,"data_compressed":5974,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4783,"internal_metadata":455273809},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002571"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":963,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-14T06:09:22.248150+0000","last_change":"2024-09-14T05:32:47.301032+0000","last_active":"2024-09-14T06:09:22.248150+0000","last_peered":"2024-09-14T06:09:22.248150+0000","last_clean":"2024-09-14T06:09:22.248150+0000","last_became_active":"2024-09-14T05:32:45.297981+0000","last_became_peered":"2024-09-14T05:32:45.297981+0000","last_unstale":"2024-09-14T06:09:22.248150+0000","last_undegraded":"2024-09-14T06:09:22.248150+0000","last_fullsized":"2024-09-14T06:09:22.248150+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_clean_scrub_stamp":"2024-09-14T05:32:44.283339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:13:28.873678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059651699999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":963,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-14T06:09:22.246531+0000","last_change":"2024-09-14T05:32:47.301315+0000","last_active":"2024-09-14T06:09:22.246531+0000","last_peered":"2024-09-14T06:09:22.246531+0000","last_clean":"2024-09-14T06:09:22.246531+0000","last_became_active":"2024-09-14T05:32:45.296192+0000","last_became_peered":"2024-09-14T05:32:45.296192+0000","last_unstale":"2024-09-14T06:09:22.246531+0000","last_undegraded":"2024-09-14T06:09:22.246531+0000","last_fullsized":"2024-09-14T06:09:22.246531+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_clean_scrub_stamp":"2024-09-14T05:32:44.283339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:48:19.029283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079042800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":963,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-14T06:09:22.247934+0000","last_change":"2024-09-14T05:32:47.301413+0000","last_active":"2024-09-14T06:09:22.247934+0000","last_peered":"2024-09-14T06:09:22.247934+0000","last_clean":"2024-09-14T06:09:22.247934+0000","last_became_active":"2024-09-14T05:32:45.296187+0000","last_became_peered":"2024-09-14T05:32:45.296187+0000","last_unstale":"2024-09-14T06:09:22.247934+0000","last_undegraded":"2024-09-14T06:09:22.247934+0000","last_fullsized":"2024-09-14T06:09:22.247934+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_clean_scrub_stamp":"2024-09-14T05:32:44.283339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:07:34.288707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089357300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":963,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-14T06:09:22.245031+0000","last_change":"2024-09-14T05:32:47.301642+0000","last_active":"2024-09-14T06:09:22.245031+0000","last_peered":"2024-09-14T06:09:22.245031+0000","last_clean":"2024-09-14T06:09:22.245031+0000","last_became_active":"2024-09-14T05:32:45.296113+0000","last_became_peered":"2024-09-14T05:32:45.296113+0000","last_unstale":"2024-09-14T06:09:22.245031+0000","last_undegraded":"2024-09-14T06:09:22.245031+0000","last_fullsized":"2024-09-14T06:09:22.245031+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_clean_scrub_stamp":"2024-09-14T05:32:44.283339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:11:57.020807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088137700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":965,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-14T06:09:21.966886+0000","last_change":"2024-09-14T05:32:47.298252+0000","last_active":"2024-09-14T06:09:21.966886+0000","last_peered":"2024-09-14T06:09:21.966886+0000","last_clean":"2024-09-14T06:09:21.966886+0000","last_became_active":"2024-09-14T05:32:45.298707+0000","last_became_peered":"2024-09-14T05:32:45.298707+0000","last_unstale":"2024-09-14T06:09:21.966886+0000","last_undegraded":"2024-09-14T06:09:21.966886+0000","last_fullsized":"2024-09-14T06:09:21.966886+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_clean_scrub_stamp":"2024-09-14T05:32:44.283339+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:20:40.111046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054093499999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":961,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-14T06:09:21.113522+0000","last_change":"2024-09-14T05:32:47.511782+0000","last_active":"2024-09-14T06:09:21.113522+0000","last_peered":"2024-09-14T06:09:21.113522+0000","last_clean":"2024-09-14T06:09:21.113522+0000","last_became_active":"2024-09-14T05:32:45.296961+0000","last_became_peered":"2024-09-14T05:32:45.296961+0000","last_unstale":"2024-09-14T06:09:21.113522+0000","last_undegraded":"2024-09-14T06:09:21.113522+0000","last_fullsized":"2024-09-14T06:09:21.113522+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_clean_scrub_stamp":"2024-09-14T05:32:44.283339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:49:29.999927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031355,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":962,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-14T06:09:21.114505+0000","last_change":"2024-09-14T05:32:47.511875+0000","last_active":"2024-09-14T06:09:21.114505+0000","last_peered":"2024-09-14T06:09:21.114505+0000","last_clean":"2024-09-14T06:09:21.114505+0000","last_became_active":"2024-09-14T05:32:45.298668+0000","last_became_peered":"2024-09-14T05:32:45.298668+0000","last_unstale":"2024-09-14T06:09:21.114505+0000","last_undegraded":"2024-09-14T06:09:21.114505+0000","last_fullsized":"2024-09-14T06:09:21.114505+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_clean_scrub_stamp":"2024-09-14T05:32:44.283339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T16:47:21.555137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035534800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":964,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-14T06:09:22.248801+0000","last_change":"2024-09-14T05:32:47.301008+0000","last_active":"2024-09-14T06:09:22.248801+0000","last_peered":"2024-09-14T06:09:22.248801+0000","last_clean":"2024-09-14T06:09:22.248801+0000","last_became_active":"2024-09-14T05:32:45.295172+0000","last_became_peered":"2024-09-14T05:32:45.295172+0000","last_unstale":"2024-09-14T06:09:22.248801+0000","last_undegraded":"2024-09-14T06:09:22.248801+0000","last_fullsized":"2024-09-14T06:09:22.248801+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_clean_scrub_stamp":"2024-09-14T05:32:44.283339+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:38:02.333228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061610699999999996,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":1031,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-14T06:09:22.247686+0000","last_change":"2024-09-14T05:32:41.276119+0000","last_active":"2024-09-14T06:09:22.247686+0000","last_peered":"2024-09-14T06:09:22.247686+0000","last_clean":"2024-09-14T06:09:22.247686+0000","last_became_active":"2024-09-14T05:32:41.275140+0000","last_became_peered":"2024-09-14T05:32:41.275140+0000","last_unstale":"2024-09-14T06:09:22.247686+0000","last_undegraded":"2024-09-14T06:09:22.247686+0000","last_fullsized":"2024-09-14T06:09:22.247686+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:32:40.262579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:32:40.262579+0000","last_clean_scrub_stamp":"2024-09-14T05:32:40.262579+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:45:44.677928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5974,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654706114,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":129504,"kb_used_data":3916,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":94242336,"statfs":{"total":96636764160,"available":96504152064,"internally_reserved":0,"allocated":4009984,"data_stored":2699422,"data_compressed":2987,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706113,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":165540,"kb_used_data":3916,"kb_used_omap":1,"kb_used_meta":161598,"kb_avail":94206300,"statfs":{"total":96636764160,"available":96467251200,"internally_reserved":0,"allocated":4009984,"data_stored":2699422,"data_compressed":2987,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":165476807},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738819,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":161320,"kb_used_data":3856,"kb_used_omap":1,"kb_used_meta":157438,"kb_avail":94210520,"statfs":{"total":96636764160,"available":96471572480,"internally_reserved":0,"allocated":3948544,"data_stored":2109035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":161216965},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2987,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2987,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T06:09:50.724 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-14T06:09:51.009 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T06:09:51.009 INFO:teuthology.orchestra.run.smithi143.stdout:{"epoch":480,"fsid":"5628b0ee-4376-46aa-afba-0841d836b50f","created":"2024-09-14T05:32:28.774394+0000","modified":"2024-09-14T06:09:21.107196+0000","last_up_change":"2024-09-14T05:32:38.257268+0000","last_in_change":"2024-09-14T05:32:29.632378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-14T05:32:39.405204+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-14T05:32:44.015925+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":17,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7cec17d-6ff1-4ab5-8f4c-ed350a9a55cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":2308157084},{"type":"v1","addr":"172.21.15.143:6817","nonce":2308157084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":2308157084},{"type":"v1","addr":"172.21.15.143:6819","nonce":2308157084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":2308157084},{"type":"v1","addr":"172.21.15.143:6823","nonce":2308157084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":2308157084},{"type":"v1","addr":"172.21.15.143:6821","nonce":2308157084}]},"public_addr":"172.21.15.143:6817/2308157084","cluster_addr":"172.21.15.143:6819/2308157084","heartbeat_back_addr":"172.21.15.143:6823/2308157084","heartbeat_front_addr":"172.21.15.143:6821/2308157084","state":["exists","up"]},{"osd":1,"uuid":"f93e9b84-2f50-46a0-9a48-e6269a52ddb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":2580893955},{"type":"v1","addr":"172.21.15.143:6809","nonce":2580893955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":2580893955},{"type":"v1","addr":"172.21.15.143:6811","nonce":2580893955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":2580893955},{"type":"v1","addr":"172.21.15.143:6815","nonce":2580893955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":2580893955},{"type":"v1","addr":"172.21.15.143:6813","nonce":2580893955}]},"public_addr":"172.21.15.143:6809/2580893955","cluster_addr":"172.21.15.143:6811/2580893955","heartbeat_back_addr":"172.21.15.143:6815/2580893955","heartbeat_front_addr":"172.21.15.143:6813/2580893955","state":["exists","up"]},{"osd":2,"uuid":"eddc7c00-4b16-4193-afa4-ab49db2d1390","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":3394389476},{"type":"v1","addr":"172.21.15.143:6801","nonce":3394389476}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":3394389476},{"type":"v1","addr":"172.21.15.143:6803","nonce":3394389476}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":3394389476},{"type":"v1","addr":"172.21.15.143:6807","nonce":3394389476}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":3394389476},{"type":"v1","addr":"172.21.15.143:6805","nonce":3394389476}]},"public_addr":"172.21.15.143:6801/3394389476","cluster_addr":"172.21.15.143:6803/3394389476","heartbeat_back_addr":"172.21.15.143:6807/3394389476","heartbeat_front_addr":"172.21.15.143:6805/3394389476","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T05:32:36.304155+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T05:32:36.349570+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-14T05:32:35.787996+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":198,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-14T06:09:52.022 INFO:tasks.ceph:Scrubbing cluster1.osd.0 2024-09-14T06:09:52.023 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-14T06:09:52.159 INFO:teuthology.orchestra.run.smithi143.stdout:{ 2024-09-14T06:09:52.159 INFO:teuthology.orchestra.run.smithi143.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-14T06:09:52.159 INFO:teuthology.orchestra.run.smithi143.stdout:} 2024-09-14T06:09:52.170 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 0 2024-09-14T06:09:52.453 INFO:teuthology.orchestra.run.smithi143.stderr:instructed osd(s) 0 to deep-scrub 2024-09-14T06:09:52.466 INFO:tasks.ceph:Scrubbing cluster1.osd.1 2024-09-14T06:09:52.466 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-14T06:09:52.590 INFO:teuthology.orchestra.run.smithi143.stdout:{ 2024-09-14T06:09:52.590 INFO:teuthology.orchestra.run.smithi143.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-14T06:09:52.590 INFO:teuthology.orchestra.run.smithi143.stdout:} 2024-09-14T06:09:52.601 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 1 2024-09-14T06:09:52.885 INFO:teuthology.orchestra.run.smithi143.stderr:instructed osd(s) 1 to deep-scrub 2024-09-14T06:09:52.898 INFO:tasks.ceph:Scrubbing cluster1.osd.2 2024-09-14T06:09:52.898 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-14T06:09:53.020 INFO:teuthology.orchestra.run.smithi143.stdout:{ 2024-09-14T06:09:53.020 INFO:teuthology.orchestra.run.smithi143.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-14T06:09:53.020 INFO:teuthology.orchestra.run.smithi143.stdout:} 2024-09-14T06:09:53.030 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 2 2024-09-14T06:09:53.315 INFO:teuthology.orchestra.run.smithi143.stderr:instructed osd(s) 2 to deep-scrub 2024-09-14T06:09:53.328 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-14T06:09:53.608 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T06:09:53.608 INFO:teuthology.orchestra.run.smithi143.stderr:dumped all 2024-09-14T06:09:53.620 INFO:teuthology.orchestra.run.smithi143.stdout:{"pg_ready":true,"pg_map":{"version":1595,"stamp":"2024-09-14T06:09:51.777534+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":456364,"kb_used_data":11688,"kb_used_omap":4,"kb_used_meta":444603,"kb_avail":282659156,"statfs":{"total":289910292480,"available":289442975744,"internally_reserved":0,"allocated":11968512,"data_stored":7507879,"data_compressed":5974,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4783,"internal_metadata":455273809},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002615"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":963,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-14T06:09:22.248150+0000","last_change":"2024-09-14T05:32:47.301032+0000","last_active":"2024-09-14T06:09:22.248150+0000","last_peered":"2024-09-14T06:09:22.248150+0000","last_clean":"2024-09-14T06:09:22.248150+0000","last_became_active":"2024-09-14T05:32:45.297981+0000","last_became_peered":"2024-09-14T05:32:45.297981+0000","last_unstale":"2024-09-14T06:09:22.248150+0000","last_undegraded":"2024-09-14T06:09:22.248150+0000","last_fullsized":"2024-09-14T06:09:22.248150+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_clean_scrub_stamp":"2024-09-14T05:32:44.283339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:13:28.873678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059651699999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":963,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-14T06:09:22.246531+0000","last_change":"2024-09-14T05:32:47.301315+0000","last_active":"2024-09-14T06:09:22.246531+0000","last_peered":"2024-09-14T06:09:22.246531+0000","last_clean":"2024-09-14T06:09:22.246531+0000","last_became_active":"2024-09-14T05:32:45.296192+0000","last_became_peered":"2024-09-14T05:32:45.296192+0000","last_unstale":"2024-09-14T06:09:22.246531+0000","last_undegraded":"2024-09-14T06:09:22.246531+0000","last_fullsized":"2024-09-14T06:09:22.246531+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_clean_scrub_stamp":"2024-09-14T05:32:44.283339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T12:48:19.029283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079042800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":963,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-14T06:09:22.247934+0000","last_change":"2024-09-14T05:32:47.301413+0000","last_active":"2024-09-14T06:09:22.247934+0000","last_peered":"2024-09-14T06:09:22.247934+0000","last_clean":"2024-09-14T06:09:22.247934+0000","last_became_active":"2024-09-14T05:32:45.296187+0000","last_became_peered":"2024-09-14T05:32:45.296187+0000","last_unstale":"2024-09-14T06:09:22.247934+0000","last_undegraded":"2024-09-14T06:09:22.247934+0000","last_fullsized":"2024-09-14T06:09:22.247934+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_clean_scrub_stamp":"2024-09-14T05:32:44.283339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:07:34.288707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089357300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":963,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-14T06:09:22.245031+0000","last_change":"2024-09-14T05:32:47.301642+0000","last_active":"2024-09-14T06:09:22.245031+0000","last_peered":"2024-09-14T06:09:22.245031+0000","last_clean":"2024-09-14T06:09:22.245031+0000","last_became_active":"2024-09-14T05:32:45.296113+0000","last_became_peered":"2024-09-14T05:32:45.296113+0000","last_unstale":"2024-09-14T06:09:22.245031+0000","last_undegraded":"2024-09-14T06:09:22.245031+0000","last_fullsized":"2024-09-14T06:09:22.245031+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_clean_scrub_stamp":"2024-09-14T05:32:44.283339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T08:11:57.020807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088137700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"17'2","reported_seq":965,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-14T06:09:21.966886+0000","last_change":"2024-09-14T05:32:47.298252+0000","last_active":"2024-09-14T06:09:21.966886+0000","last_peered":"2024-09-14T06:09:21.966886+0000","last_clean":"2024-09-14T06:09:21.966886+0000","last_became_active":"2024-09-14T05:32:45.298707+0000","last_became_peered":"2024-09-14T05:32:45.298707+0000","last_unstale":"2024-09-14T06:09:21.966886+0000","last_undegraded":"2024-09-14T06:09:21.966886+0000","last_fullsized":"2024-09-14T06:09:21.966886+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_clean_scrub_stamp":"2024-09-14T05:32:44.283339+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:20:40.111046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054093499999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":961,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-14T06:09:21.113522+0000","last_change":"2024-09-14T05:32:47.511782+0000","last_active":"2024-09-14T06:09:21.113522+0000","last_peered":"2024-09-14T06:09:21.113522+0000","last_clean":"2024-09-14T06:09:21.113522+0000","last_became_active":"2024-09-14T05:32:45.296961+0000","last_became_peered":"2024-09-14T05:32:45.296961+0000","last_unstale":"2024-09-14T06:09:21.113522+0000","last_undegraded":"2024-09-14T06:09:21.113522+0000","last_fullsized":"2024-09-14T06:09:21.113522+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_clean_scrub_stamp":"2024-09-14T05:32:44.283339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:49:29.999927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031355,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":962,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-14T06:09:21.114505+0000","last_change":"2024-09-14T05:32:47.511875+0000","last_active":"2024-09-14T06:09:21.114505+0000","last_peered":"2024-09-14T06:09:21.114505+0000","last_clean":"2024-09-14T06:09:21.114505+0000","last_became_active":"2024-09-14T05:32:45.298668+0000","last_became_peered":"2024-09-14T05:32:45.298668+0000","last_unstale":"2024-09-14T06:09:21.114505+0000","last_undegraded":"2024-09-14T06:09:21.114505+0000","last_fullsized":"2024-09-14T06:09:21.114505+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_clean_scrub_stamp":"2024-09-14T05:32:44.283339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T16:47:21.555137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035534800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"15'1","reported_seq":964,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-14T06:09:22.248801+0000","last_change":"2024-09-14T05:32:47.301008+0000","last_active":"2024-09-14T06:09:22.248801+0000","last_peered":"2024-09-14T06:09:22.248801+0000","last_clean":"2024-09-14T06:09:22.248801+0000","last_became_active":"2024-09-14T05:32:45.295172+0000","last_became_peered":"2024-09-14T05:32:45.295172+0000","last_unstale":"2024-09-14T06:09:22.248801+0000","last_undegraded":"2024-09-14T06:09:22.248801+0000","last_fullsized":"2024-09-14T06:09:22.248801+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:32:44.283339+0000","last_clean_scrub_stamp":"2024-09-14T05:32:44.283339+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:38:02.333228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061610699999999996,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":1031,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-14T06:09:22.247686+0000","last_change":"2024-09-14T05:32:41.276119+0000","last_active":"2024-09-14T06:09:22.247686+0000","last_peered":"2024-09-14T06:09:22.247686+0000","last_clean":"2024-09-14T06:09:22.247686+0000","last_became_active":"2024-09-14T05:32:41.275140+0000","last_became_peered":"2024-09-14T05:32:41.275140+0000","last_unstale":"2024-09-14T06:09:22.247686+0000","last_undegraded":"2024-09-14T06:09:22.247686+0000","last_fullsized":"2024-09-14T06:09:22.247686+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T05:32:40.262579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T05:32:40.262579+0000","last_clean_scrub_stamp":"2024-09-14T05:32:40.262579+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:45:44.677928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5974,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654706114,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":129504,"kb_used_data":3916,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":94242336,"statfs":{"total":96636764160,"available":96504152064,"internally_reserved":0,"allocated":4009984,"data_stored":2699422,"data_compressed":2987,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706113,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":165540,"kb_used_data":3916,"kb_used_omap":1,"kb_used_meta":161598,"kb_avail":94206300,"statfs":{"total":96636764160,"available":96467251200,"internally_reserved":0,"allocated":4009984,"data_stored":2699422,"data_compressed":2987,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":165476807},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738819,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":161320,"kb_used_data":3856,"kb_used_omap":1,"kb_used_meta":157438,"kb_avail":94210520,"statfs":{"total":96636764160,"available":96471572480,"internally_reserved":0,"allocated":3948544,"data_stored":2109035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":161216965},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2987,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2987,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T06:09:53.621 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-14T05:32:44.283339+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=5, tm_min=32, tm_sec=44, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=6, tm_min=9, tm_sec=51, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T06:09:53.621 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-14T05:32:44.283339+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=5, tm_min=32, tm_sec=44, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=6, tm_min=9, tm_sec=51, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T06:09:53.621 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-14T05:32:44.283339+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=5, tm_min=32, tm_sec=44, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=6, tm_min=9, tm_sec=51, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T06:09:53.622 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-14T05:32:44.283339+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=5, tm_min=32, tm_sec=44, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=6, tm_min=9, tm_sec=51, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T06:09:53.622 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-14T05:32:44.283339+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=5, tm_min=32, tm_sec=44, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=6, tm_min=9, tm_sec=51, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T06:09:53.622 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-14T05:32:44.283339+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=5, tm_min=32, tm_sec=44, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=6, tm_min=9, tm_sec=51, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T06:09:53.622 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-14T05:32:44.283339+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=5, tm_min=32, tm_sec=44, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=6, tm_min=9, tm_sec=51, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T06:09:53.622 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-14T05:32:44.283339+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=5, tm_min=32, tm_sec=44, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=6, tm_min=9, tm_sec=51, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T06:09:53.622 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-14T05:32:40.262579+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=5, tm_min=32, tm_sec=40, tm_wday=5, tm_yday=258, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=14, tm_hour=6, tm_min=9, tm_sec=51, tm_wday=5, tm_yday=258, tm_isdst=0) 2024-09-14T06:09:53.622 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-14T06:10:13.623 DEBUG:teuthology.orchestra.run.smithi143:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-14T06:10:13.919 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-09-14T06:10:13.920 INFO:teuthology.orchestra.run.smithi143.stderr:dumped all 2024-09-14T06:10:13.932 INFO:teuthology.orchestra.run.smithi143.stdout:{"pg_ready":true,"pg_map":{"version":1606,"stamp":"2024-09-14T06:10:13.781932+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":456344,"kb_used_data":11688,"kb_used_omap":4,"kb_used_meta":444603,"kb_avail":282659176,"statfs":{"total":289910292480,"available":289442996224,"internally_reserved":0,"allocated":11968512,"data_stored":7507879,"data_compressed":5974,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4783,"internal_metadata":455273809},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002278"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":971,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-14T06:09:56.721674+0000","last_change":"2024-09-14T06:09:56.721674+0000","last_active":"2024-09-14T06:09:56.721674+0000","last_peered":"2024-09-14T06:09:56.721674+0000","last_clean":"2024-09-14T06:09:56.721674+0000","last_became_active":"2024-09-14T05:32:45.297981+0000","last_became_peered":"2024-09-14T05:32:45.297981+0000","last_unstale":"2024-09-14T06:09:56.721674+0000","last_undegraded":"2024-09-14T06:09:56.721674+0000","last_fullsized":"2024-09-14T06:09:56.721674+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:09:56.721571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:09:56.721571+0000","last_clean_scrub_stamp":"2024-09-14T06:09:56.721571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T17:32:42.749278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059651699999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":971,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-14T06:09:55.705676+0000","last_change":"2024-09-14T06:09:55.705676+0000","last_active":"2024-09-14T06:09:55.705676+0000","last_peered":"2024-09-14T06:09:55.705676+0000","last_clean":"2024-09-14T06:09:55.705676+0000","last_became_active":"2024-09-14T05:32:45.296192+0000","last_became_peered":"2024-09-14T05:32:45.296192+0000","last_unstale":"2024-09-14T06:09:55.705676+0000","last_undegraded":"2024-09-14T06:09:55.705676+0000","last_fullsized":"2024-09-14T06:09:55.705676+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:09:55.705591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:09:55.705591+0000","last_clean_scrub_stamp":"2024-09-14T06:09:55.705591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:01:15.723786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079042800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":971,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-14T06:09:58.757474+0000","last_change":"2024-09-14T06:09:58.757474+0000","last_active":"2024-09-14T06:09:58.757474+0000","last_peered":"2024-09-14T06:09:58.757474+0000","last_clean":"2024-09-14T06:09:58.757474+0000","last_became_active":"2024-09-14T05:32:45.296187+0000","last_became_peered":"2024-09-14T05:32:45.296187+0000","last_unstale":"2024-09-14T06:09:58.757474+0000","last_undegraded":"2024-09-14T06:09:58.757474+0000","last_fullsized":"2024-09-14T06:09:58.757474+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:09:58.757345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:09:58.757345+0000","last_clean_scrub_stamp":"2024-09-14T06:09:58.757345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T11:51:34.001166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089357300000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":971,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-14T06:09:54.737322+0000","last_change":"2024-09-14T06:09:54.737322+0000","last_active":"2024-09-14T06:09:54.737322+0000","last_peered":"2024-09-14T06:09:54.737322+0000","last_clean":"2024-09-14T06:09:54.737322+0000","last_became_active":"2024-09-14T05:32:45.296113+0000","last_became_peered":"2024-09-14T05:32:45.296113+0000","last_unstale":"2024-09-14T06:09:54.737322+0000","last_undegraded":"2024-09-14T06:09:54.737322+0000","last_fullsized":"2024-09-14T06:09:54.737322+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:09:54.737247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:09:54.737247+0000","last_clean_scrub_stamp":"2024-09-14T06:09:54.737247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T16:52:33.968150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088137700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"480'3","reported_seq":974,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-14T06:09:53.044723+0000","last_change":"2024-09-14T06:09:53.044723+0000","last_active":"2024-09-14T06:09:53.044723+0000","last_peered":"2024-09-14T06:09:53.044723+0000","last_clean":"2024-09-14T06:09:53.044723+0000","last_became_active":"2024-09-14T05:32:45.298707+0000","last_became_peered":"2024-09-14T05:32:45.298707+0000","last_unstale":"2024-09-14T06:09:53.044723+0000","last_undegraded":"2024-09-14T06:09:53.044723+0000","last_fullsized":"2024-09-14T06:09:53.044723+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'3","last_scrub_stamp":"2024-09-14T06:09:53.044649+0000","last_deep_scrub":"480'3","last_deep_scrub_stamp":"2024-09-14T06:09:53.044649+0000","last_clean_scrub_stamp":"2024-09-14T06:09:53.044649+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T14:10:09.434568+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00054093499999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":969,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-14T06:09:54.255980+0000","last_change":"2024-09-14T06:09:54.255980+0000","last_active":"2024-09-14T06:09:54.255980+0000","last_peered":"2024-09-14T06:09:54.255980+0000","last_clean":"2024-09-14T06:09:54.255980+0000","last_became_active":"2024-09-14T05:32:45.296961+0000","last_became_peered":"2024-09-14T05:32:45.296961+0000","last_unstale":"2024-09-14T06:09:54.255980+0000","last_undegraded":"2024-09-14T06:09:54.255980+0000","last_fullsized":"2024-09-14T06:09:54.255980+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:09:54.255911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:09:54.255911+0000","last_clean_scrub_stamp":"2024-09-14T06:09:54.255911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T16:38:21.273271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031355,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":970,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-14T06:09:55.212846+0000","last_change":"2024-09-14T06:09:55.212846+0000","last_active":"2024-09-14T06:09:55.212846+0000","last_peered":"2024-09-14T06:09:55.212846+0000","last_clean":"2024-09-14T06:09:55.212846+0000","last_became_active":"2024-09-14T05:32:45.298668+0000","last_became_peered":"2024-09-14T05:32:45.298668+0000","last_unstale":"2024-09-14T06:09:55.212846+0000","last_undegraded":"2024-09-14T06:09:55.212846+0000","last_fullsized":"2024-09-14T06:09:55.212846+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-14T06:09:55.212750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-14T06:09:55.212750+0000","last_clean_scrub_stamp":"2024-09-14T06:09:55.212750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T10:35:53.155752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035534800000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"480'2","reported_seq":973,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-14T06:09:53.775376+0000","last_change":"2024-09-14T06:09:53.775376+0000","last_active":"2024-09-14T06:09:53.775376+0000","last_peered":"2024-09-14T06:09:53.775376+0000","last_clean":"2024-09-14T06:09:53.775376+0000","last_became_active":"2024-09-14T05:32:45.295172+0000","last_became_peered":"2024-09-14T05:32:45.295172+0000","last_unstale":"2024-09-14T06:09:53.775376+0000","last_undegraded":"2024-09-14T06:09:53.775376+0000","last_fullsized":"2024-09-14T06:09:53.775376+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'2","last_scrub_stamp":"2024-09-14T06:09:53.775305+0000","last_deep_scrub":"480'2","last_deep_scrub_stamp":"2024-09-14T06:09:53.775305+0000","last_clean_scrub_stamp":"2024-09-14T06:09:53.775305+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T07:43:56.786201+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00061610699999999996,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"480'54","reported_seq":1041,"reported_epoch":480,"state":"active+clean","last_fresh":"2024-09-14T06:09:57.744495+0000","last_change":"2024-09-14T06:09:57.744495+0000","last_active":"2024-09-14T06:09:57.744495+0000","last_peered":"2024-09-14T06:09:57.744495+0000","last_clean":"2024-09-14T06:09:57.744495+0000","last_became_active":"2024-09-14T05:32:41.275140+0000","last_became_peered":"2024-09-14T05:32:41.275140+0000","last_unstale":"2024-09-14T06:09:57.744495+0000","last_undegraded":"2024-09-14T06:09:57.744495+0000","last_fullsized":"2024-09-14T06:09:57.744495+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'54","last_scrub_stamp":"2024-09-14T06:09:57.744383+0000","last_deep_scrub":"480'54","last_deep_scrub_stamp":"2024-09-14T06:09:57.744383+0000","last_clean_scrub_stamp":"2024-09-14T06:09:57.744383+0000","objects_scrubbed":2,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T09:10:53.813990+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5974,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654706118,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":129500,"kb_used_data":3916,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":94242340,"statfs":{"total":96636764160,"available":96504156160,"internally_reserved":0,"allocated":4009984,"data_stored":2699422,"data_compressed":2987,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654706117,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":165532,"kb_used_data":3916,"kb_used_omap":1,"kb_used_meta":161598,"kb_avail":94206308,"statfs":{"total":96636764160,"available":96467259392,"internally_reserved":0,"allocated":4009984,"data_stored":2699422,"data_compressed":2987,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":165476807},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738824,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":161312,"kb_used_data":3856,"kb_used_omap":1,"kb_used_meta":157438,"kb_avail":94210528,"statfs":{"total":96636764160,"available":96471580672,"internally_reserved":0,"allocated":3948544,"data_stored":2109035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":161216965},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2987,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2987,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-14T06:10:13.933 DEBUG:teuthology.orchestra.run.smithi143:> sudo ceph --cluster cluster1 config set global mon_health_to_clog false 2024-09-14T06:10:14.232 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-14T06:10:14.232 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-14T06:10:14.233 DEBUG:tasks.ceph.cluster1.osd.0:waiting for process to exit 2024-09-14T06:10:14.233 INFO:teuthology.orchestra.run:waiting for 300 2024-09-14T06:10:14.284 INFO:tasks.ceph.cluster1.osd.0:Stopped 2024-09-14T06:10:14.285 DEBUG:tasks.ceph.cluster1.osd.1:waiting for process to exit 2024-09-14T06:10:14.285 INFO:teuthology.orchestra.run:waiting for 300 2024-09-14T06:10:14.331 INFO:tasks.ceph.cluster1.osd.1:Stopped 2024-09-14T06:10:14.331 DEBUG:tasks.ceph.cluster1.osd.2:waiting for process to exit 2024-09-14T06:10:14.331 INFO:teuthology.orchestra.run:waiting for 300 2024-09-14T06:10:14.381 INFO:tasks.ceph.cluster1.osd.2:Stopped 2024-09-14T06:10:14.381 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-14T06:10:14.381 DEBUG:tasks.ceph.cluster1.mgr.x:waiting for process to exit 2024-09-14T06:10:14.381 INFO:teuthology.orchestra.run:waiting for 300 2024-09-14T06:10:14.408 INFO:tasks.ceph.cluster1.mgr.x:Stopped 2024-09-14T06:10:14.408 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-14T06:10:14.408 DEBUG:tasks.ceph.cluster1.mon.a:waiting for process to exit 2024-09-14T06:10:14.408 INFO:teuthology.orchestra.run:waiting for 300 2024-09-14T06:10:14.460 INFO:tasks.ceph.cluster1.mon.a:Stopped 2024-09-14T06:10:14.461 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-14T06:10:14.461 DEBUG:teuthology.orchestra.run.smithi143:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-14T06:10:14.527 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-0 on ubuntu@smithi143.front.sepia.ceph.com 2024-09-14T06:10:14.527 DEBUG:teuthology.orchestra.run.smithi143:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-0 2024-09-14T06:10:14.761 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-1 on ubuntu@smithi143.front.sepia.ceph.com 2024-09-14T06:10:14.761 DEBUG:teuthology.orchestra.run.smithi143:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-1 2024-09-14T06:10:14.817 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-2 on ubuntu@smithi143.front.sepia.ceph.com 2024-09-14T06:10:14.818 DEBUG:teuthology.orchestra.run.smithi143:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-2 2024-09-14T06:10:14.887 INFO:tasks.ceph:Archiving mon data... 2024-09-14T06:10:14.887 DEBUG:teuthology.misc:Transferring archived files from smithi143:/var/lib/ceph/mon/cluster1-a to /home/teuthworker/archive/teuthology-2024-09-10_20:16:17-rbd-main-distro-default-smithi/7900324/data/cluster1.mon.a.tgz 2024-09-14T06:10:14.887 DEBUG:teuthology.orchestra.run.smithi143:> mktemp 2024-09-14T06:10:14.903 INFO:teuthology.orchestra.run.smithi143.stdout:/tmp/tmp.bXqTx0Izy9 2024-09-14T06:10:14.903 DEBUG:teuthology.orchestra.run.smithi143:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster1-a -- . > /tmp/tmp.bXqTx0Izy9 2024-09-14T06:10:15.067 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 0666 /tmp/tmp.bXqTx0Izy9 2024-09-14T06:10:15.150 DEBUG:teuthology.orchestra.remote:smithi143:/tmp/tmp.bXqTx0Izy9 is 407KB 2024-09-14T06:10:15.246 DEBUG:teuthology.orchestra.run.smithi143:> rm -fr /tmp/tmp.bXqTx0Izy9 2024-09-14T06:10:15.261 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-14T06:10:15.261 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-09-14T06:10:15.303 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-09-14T06:10:15.382 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-14T06:10:15.401 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-14T06:10:15.401 INFO:tasks.ceph:Archiving crash dumps... 2024-09-14T06:10:15.403 DEBUG:teuthology.misc:Transferring archived files from smithi143:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-10_20:16:17-rbd-main-distro-default-smithi/7900324/remote/smithi143/crash 2024-09-14T06:10:15.404 DEBUG:teuthology.orchestra.run.smithi143:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-14T06:10:15.447 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-10_20:16:17-rbd-main-distro-default-smithi/7900324/remote/smithi191/crash 2024-09-14T06:10:15.448 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-14T06:10:15.477 INFO:tasks.ceph:Compressing logs... 2024-09-14T06:10:15.477 DEBUG:teuthology.orchestra.run.smithi143:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-14T06:10:15.492 DEBUG:teuthology.orchestra.run.smithi191:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-14T06:10:15.514 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37035.log 2024-09-14T06:10:15.514 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35950.log 2024-09-14T06:10:15.514 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37413.log 2024-09-14T06:10:15.532 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37161.log 2024-09-14T06:10:15.532 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37203.log 2024-09-14T06:10:15.532 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/cluster1-client.admin.37035.log: /var/log/ceph/cluster1-client.admin.35950.log: /var/log/ceph/cluster1-client.admin.37413.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37035.log.gz 2024-09-14T06:10:15.532 INFO:teuthology.orchestra.run.smithi143.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35950.log.gz 2024-09-14T06:10:15.532 INFO:teuthology.orchestra.run.smithi143.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37413.log.gz 2024-09-14T06:10:15.532 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-mon.a.log 2024-09-14T06:10:15.532 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/cluster1-client.admin.37161.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37161.log.gz 2024-09-14T06:10:15.532 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/cluster1-client.admin.37203.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32232.log 2024-09-14T06:10:15.532 INFO:teuthology.orchestra.run.smithi143.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37203.log.gz 2024-09-14T06:10:15.533 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/cluster1-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37077.log 2024-09-14T06:10:15.533 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/cluster1-client.admin.32232.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37119.log 2024-09-14T06:10:15.533 INFO:teuthology.orchestra.run.smithi143.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32232.log.gz 2024-09-14T06:10:15.533 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/cluster1-client.admin.37077.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35446.log 2024-09-14T06:10:15.533 INFO:teuthology.orchestra.run.smithi143.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37077.log.gz 2024-09-14T06:10:15.533 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/cluster1-client.admin.37119.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35842.log 2024-09-14T06:10:15.533 INFO:teuthology.orchestra.run.smithi143.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37119.log.gz 2024-09-14T06:10:15.533 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/cluster1-client.admin.35446.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.1.log 2024-09-14T06:10:15.533 INFO:teuthology.orchestra.run.smithi143.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35446.log.gz 2024-09-14T06:10:15.533 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/cluster1-client.admin.35842.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35065.log 2024-09-14T06:10:15.533 INFO:teuthology.orchestra.run.smithi143.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35842.log.gz 2024-09-14T06:10:15.533 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/cluster1-osd.1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37371.log 2024-09-14T06:10:15.533 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/cluster1-client.admin.35065.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35065.log.gz 2024-09-14T06:10:15.534 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37498.log 2024-09-14T06:10:15.534 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35362.log 2024-09-14T06:10:15.534 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/cluster1-client.admin.37371.log: /var/log/ceph/cluster1-client.admin.37498.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37371.log.gz 2024-09-14T06:10:15.534 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37287.log 2024-09-14T06:10:15.534 INFO:teuthology.orchestra.run.smithi143.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37498.log.gz 2024-09-14T06:10:15.534 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/cluster1-client.admin.35362.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35362.log.gz 2024-09-14T06:10:15.534 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37329.log 2024-09-14T06:10:15.534 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35311.log 2024-09-14T06:10:15.534 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/cluster1-client.admin.37287.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37287.log.gz 2024-09-14T06:10:15.534 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/cluster1-client.admin.37329.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35908.log 2024-09-14T06:10:15.534 INFO:teuthology.orchestra.run.smithi143.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37329.log.gz 2024-09-14T06:10:15.534 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/cluster1-client.admin.35311.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.2.log 2024-09-14T06:10:15.534 INFO:teuthology.orchestra.run.smithi143.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.admin.35311.log.gz 2024-09-14T06:10:15.534 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37245.log 2024-09-14T06:10:15.535 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/cluster1-client.admin.35908.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35908.log.gz 2024-09-14T06:10:15.535 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/cluster1-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35802.log 2024-09-14T06:10:15.535 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/cluster1-client.admin.37245.log: gzip -5 --verbose -- /var/log/ceph/cluster2-mgr.x.log 2024-09-14T06:10:15.535 INFO:teuthology.orchestra.run.smithi143.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37245.log.gz 2024-09-14T06:10:15.535 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27594.log 2024-09-14T06:10:15.535 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/cluster1-client.admin.35802.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35802.log.gz 2024-09-14T06:10:15.535 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/cluster2-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32475.log 2024-09-14T06:10:15.535 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/ceph.tmp-client.admin.27594.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27594.log.gz 2024-09-14T06:10:15.535 INFO:teuthology.orchestra.run.smithi143.stderr: --verbose -- /var/log/ceph/cluster1-client.admin.35782.log 2024-09-14T06:10:15.535 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/cluster1-client.admin.32475.log: 25.4% -- replaced with /var/log/ceph/cluster1-client.admin.32475.log.gz 2024-09-14T06:10:15.535 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35992.log 2024-09-14T06:10:15.535 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/cluster1-client.admin.35782.log: gzip 0.0% -5 --verbose -- /var/log/ceph/cluster1-client.admin.35569.log 2024-09-14T06:10:15.535 INFO:teuthology.orchestra.run.smithi143.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.35782.log.gz 2024-09-14T06:10:15.535 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/cluster1-client.admin.35992.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32272.log 2024-09-14T06:10:15.535 INFO:teuthology.orchestra.run.smithi143.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35992.log.gz 2024-09-14T06:10:15.536 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.admin.35569.log: -- /var/log/ceph/cluster1-client.admin.35286.log 2024-09-14T06:10:15.536 INFO:teuthology.orchestra.run.smithi143.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35569.log.gz 2024-09-14T06:10:15.536 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.admin.32272.log: -- /var/log/ceph/cluster1-client.admin.35696.log 2024-09-14T06:10:15.536 INFO:teuthology.orchestra.run.smithi143.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32272.log.gz 2024-09-14T06:10:15.536 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/cluster1-client.admin.35286.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35286.log.gz 2024-09-14T06:10:15.536 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-mgr.x.log 2024-09-14T06:10:15.536 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/cluster1-client.admin.35696.log: gzip -5 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35696.log.gz 2024-09-14T06:10:15.536 INFO:teuthology.orchestra.run.smithi143.stderr: --verbose -- /var/log/ceph/cluster1-client.admin.32312.log 2024-09-14T06:10:15.536 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/cluster1-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35568.log 2024-09-14T06:10:15.536 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/cluster1-client.admin.32312.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32411.log 2024-09-14T06:10:15.536 INFO:teuthology.orchestra.run.smithi143.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.32312.log.gz 2024-09-14T06:10:15.536 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/cluster1-client.admin.35568.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35230.log 2024-09-14T06:10:15.536 INFO:teuthology.orchestra.run.smithi143.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35568.log.gz 2024-09-14T06:10:15.537 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/cluster1-client.admin.32411.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32411.log.gz 2024-09-14T06:10:15.537 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1.audit.log 2024-09-14T06:10:15.537 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/cluster1-client.admin.35230.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35230.log.gz 2024-09-14T06:10:15.537 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37455.log 2024-09-14T06:10:15.537 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-osd.0.log 2024-09-14T06:10:15.537 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/cluster1-client.admin.37455.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35656.log 2024-09-14T06:10:15.538 INFO:teuthology.orchestra.run.smithi143.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.37455.log.gz 2024-09-14T06:10:15.541 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/cluster1-osd.0.log: /var/log/ceph/cluster1.audit.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35695.log 2024-09-14T06:10:15.541 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/cluster1-client.admin.35656.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35656.log.gz 2024-09-14T06:10:15.541 INFO:teuthology.orchestra.run.smithi143.stderr: 93.0% -- replaced with /var/log/ceph/cluster1.audit.log.gz 2024-09-14T06:10:15.553 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35404.log 2024-09-14T06:10:15.554 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/cluster1-client.admin.35695.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35695.log.gz 2024-09-14T06:10:15.562 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35570.log 2024-09-14T06:10:15.562 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/cluster1-client.admin.35404.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35404.log.gz 2024-09-14T06:10:15.562 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35488.log 2024-09-14T06:10:15.562 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/cluster1-client.admin.35570.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35570.log.gz 2024-09-14T06:10:15.562 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32074.log 2024-09-14T06:10:15.562 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/cluster1-client.admin.35488.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35488.log.gz 2024-09-14T06:10:15.562 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1.log 2024-09-14T06:10:15.562 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/cluster1-client.admin.32074.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32074.log.gz 2024-09-14T06:10:15.562 INFO:teuthology.orchestra.run.smithi143.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37538.log 2024-09-14T06:10:15.571 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/cluster1.log: 87.7% -- replaced with /var/log/ceph/cluster1.log.gz 2024-09-14T06:10:15.571 INFO:teuthology.orchestra.run.smithi143.stderr:/var/log/ceph/cluster1-client.admin.37538.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37538.log.gz 2024-09-14T06:10:15.639 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43896.log 2024-09-14T06:10:15.639 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50421.log 2024-09-14T06:10:15.639 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.43896.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50039.log 2024-09-14T06:10:15.639 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.50421.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58901.log 2024-09-14T06:10:15.639 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.50039.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44392.log 2024-09-14T06:10:15.640 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.50039.log.gz 2024-09-14T06:10:15.640 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.58901.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68431.log 2024-09-14T06:10:15.640 INFO:teuthology.orchestra.run.smithi191.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50421.log.gz 2024-09-14T06:10:15.640 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58901.log.gz 2024-09-14T06:10:15.640 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.44392.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70052.log 2024-09-14T06:10:15.640 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.68431.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67548.log 2024-09-14T06:10:15.640 INFO:teuthology.orchestra.run.smithi191.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.44392.log.gz 2024-09-14T06:10:15.640 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.70052.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35735.log 2024-09-14T06:10:15.640 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.39996.log 2024-09-14T06:10:15.640 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.67548.log: /var/log/ceph/cluster2-client.admin.35735.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67548.log.gz 2024-09-14T06:10:15.640 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35735.log.gz 2024-09-14T06:10:15.640 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69436.log 2024-09-14T06:10:15.640 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69822.log 2024-09-14T06:10:15.641 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.3.39996.log: /var/log/ceph/cluster2-client.mirror.69436.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62953.log 2024-09-14T06:10:15.641 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.69822.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41413.log 2024-09-14T06:10:15.641 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.69822.log.gz 2024-09-14T06:10:15.641 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.62953.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48640.log 2024-09-14T06:10:15.641 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.41413.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40648.log 2024-09-14T06:10:15.641 INFO:teuthology.orchestra.run.smithi191.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43896.log.gz 2024-09-14T06:10:15.641 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48640.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71576.log 2024-09-14T06:10:15.641 INFO:teuthology.orchestra.run.smithi191.stderr: 91.1% -- replaced with /var/log/ceph/cluster2-client.mirror.69436.log.gz 87.4% 2024-09-14T06:10:15.641 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.48640.log.gz 2024-09-14T06:10:15.641 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.40648.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41110.log 2024-09-14T06:10:15.641 INFO:teuthology.orchestra.run.smithi191.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41413.log.gz 2024-09-14T06:10:15.641 INFO:teuthology.orchestra.run.smithi191.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.40648.log.gz 2024-09-14T06:10:15.641 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.71576.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55672.log 2024-09-14T06:10:15.642 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.41110.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.71576.log.gz 2024-09-14T06:10:15.642 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50465.log 2024-09-14T06:10:15.642 INFO:teuthology.orchestra.run.smithi191.stderr: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.41110.log.gz 90.3% 2024-09-14T06:10:15.642 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.62953.log.gz 2024-09-14T06:10:15.642 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.55672.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62601.log 2024-09-14T06:10:15.642 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.50465.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68223.log 2024-09-14T06:10:15.642 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.50465.log.gz 2024-09-14T06:10:15.642 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.62601.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64200.log 2024-09-14T06:10:15.642 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65498.log 2024-09-14T06:10:15.642 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.68223.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62601.log.gz 2024-09-14T06:10:15.642 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64200.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68223.log.gz 2024-09-14T06:10:15.642 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56620.log 2024-09-14T06:10:15.642 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48275.log 2024-09-14T06:10:15.642 INFO:teuthology.orchestra.run.smithi191.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.55672.log.gz 2024-09-14T06:10:15.643 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65498.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64200.log.gz 2024-09-14T06:10:15.643 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.56620.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41709.log 2024-09-14T06:10:15.643 INFO:teuthology.orchestra.run.smithi191.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65498.log.gz 2024-09-14T06:10:15.643 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48275.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55989.log 2024-09-14T06:10:15.643 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63931.log 2024-09-14T06:10:15.643 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.41709.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.56620.log.gz 2024-09-14T06:10:15.643 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.55989.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36874.log 2024-09-14T06:10:15.643 INFO:teuthology.orchestra.run.smithi191.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48275.log.gz 2024-09-14T06:10:15.643 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4%/var/log/ceph/cluster2-client.mirror.63931.log: -- replaced with /var/log/ceph/cluster2-client.mirror.55989.log.gz 2024-09-14T06:10:15.643 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56403.log 2024-09-14T06:10:15.643 INFO:teuthology.orchestra.run.smithi191.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.41709.log.gz 2024-09-14T06:10:15.643 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.36874.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61816.log 2024-09-14T06:10:15.643 INFO:teuthology.orchestra.run.smithi191.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63931.log.gz 2024-09-14T06:10:15.643 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58532.log 2024-09-14T06:10:15.644 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.56403.log: /var/log/ceph/cluster1-client.mirror.61816.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71890.log 2024-09-14T06:10:15.644 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.58532.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52560.log 2024-09-14T06:10:15.644 INFO:teuthology.orchestra.run.smithi191.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.56403.log.gz 2024-09-14T06:10:15.644 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.71890.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59360.log 2024-09-14T06:10:15.644 INFO:teuthology.orchestra.run.smithi191.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.71890.log.gz 2024-09-14T06:10:15.644 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.52560.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49439.log 2024-09-14T06:10:15.644 INFO:teuthology.orchestra.run.smithi191.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61816.log.gz 2024-09-14T06:10:15.644 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.59360.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66648.log 2024-09-14T06:10:15.644 INFO:teuthology.orchestra.run.smithi191.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.59360.log.gz 2024-09-14T06:10:15.644 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49439.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72355.log 2024-09-14T06:10:15.644 INFO:teuthology.orchestra.run.smithi191.stderr: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49439.log.gz 2024-09-14T06:10:15.644 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.66648.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64096.log 2024-09-14T06:10:15.644 INFO:teuthology.orchestra.run.smithi191.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66648.log.gz 2024-09-14T06:10:15.645 INFO:teuthology.orchestra.run.smithi191.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52560.log.gz 2024-09-14T06:10:15.645 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.72355.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43122.log 2024-09-14T06:10:15.645 INFO:teuthology.orchestra.run.smithi191.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58532.log.gz 2024-09-14T06:10:15.645 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.72355.log.gz 2024-09-14T06:10:15.645 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64096.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60255.log 2024-09-14T06:10:15.645 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.43122.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60737.log 2024-09-14T06:10:15.645 INFO:teuthology.orchestra.run.smithi191.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.43122.log.gz 2024-09-14T06:10:15.645 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.60255.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45153.log 2024-09-14T06:10:15.645 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52648.log 2024-09-14T06:10:15.645 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.60737.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.60255.log.gz 2024-09-14T06:10:15.645 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45153.log: gzip -5 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60737.log.gz 2024-09-14T06:10:15.645 INFO:teuthology.orchestra.run.smithi191.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.65707.log 2024-09-14T06:10:15.645 INFO:teuthology.orchestra.run.smithi191.stderr: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45153.log.gz 2024-09-14T06:10:15.646 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.52648.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40357.log 2024-09-14T06:10:15.646 INFO:teuthology.orchestra.run.smithi191.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64096.log.gz 2024-09-14T06:10:15.646 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65707.log: gzip 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52648.log.gz -5 2024-09-14T06:10:15.646 INFO:teuthology.orchestra.run.smithi191.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.65250.log 2024-09-14T06:10:15.646 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.40357.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68949.log 2024-09-14T06:10:15.646 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65250.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65291.log 2024-09-14T06:10:15.646 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.68949.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62322.log 2024-09-14T06:10:15.646 INFO:teuthology.orchestra.run.smithi191.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65250.log.gz 2024-09-14T06:10:15.646 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65291.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58250.log 2024-09-14T06:10:15.646 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.62322.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65728.log 2024-09-14T06:10:15.646 INFO:teuthology.orchestra.run.smithi191.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65707.log.gz 2024-09-14T06:10:15.646 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.58250.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53797.log 2024-09-14T06:10:15.646 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58250.log.gz 2024-09-14T06:10:15.647 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65728.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58274.log 2024-09-14T06:10:15.647 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.53797.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57445.log 2024-09-14T06:10:15.647 INFO:teuthology.orchestra.run.smithi191.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65728.log.gz 2024-09-14T06:10:15.647 INFO:teuthology.orchestra.run.smithi191.stderr: 92.4% 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62322.log.gz 2024-09-14T06:10:15.647 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.65291.log.gz 2024-09-14T06:10:15.647 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37300.log 2024-09-14T06:10:15.647 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.58274.log: /var/log/ceph/cluster1-client.mirror.57445.log: 89.7%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52873.log 2024-09-14T06:10:15.647 INFO:teuthology.orchestra.run.smithi191.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.58274.log.gz 2024-09-14T06:10:15.647 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.53797.log.gz 2024-09-14T06:10:15.647 INFO:teuthology.orchestra.run.smithi191.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.57445.log.gz 2024-09-14T06:10:15.647 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.37300.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40315.log 2024-09-14T06:10:15.647 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.52873.log: 53.1% -- replaced with /var/log/ceph/cluster1-client.mirror.37300.log.gz 2024-09-14T06:10:15.647 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38627.log 2024-09-14T06:10:15.648 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.40315.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38583.log 2024-09-14T06:10:15.648 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.38627.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.36874.log 2024-09-14T06:10:15.648 INFO:teuthology.orchestra.run.smithi191.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52873.log.gz 2024-09-14T06:10:15.648 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.38583.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58154.log 2024-09-14T06:10:15.648 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70875.log 2024-09-14T06:10:15.648 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.0.36874.log: /var/log/ceph/cluster2-client.mirror.58154.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45537.log 2024-09-14T06:10:15.648 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58154.log.gz 2024-09-14T06:10:15.648 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.70875.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60508.log 2024-09-14T06:10:15.648 INFO:teuthology.orchestra.run.smithi191.stderr: 81.5% -- replaced with /var/log/ceph/cluster2-client.mirror.70875.log.gz 2024-09-14T06:10:15.648 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45537.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66468.log 2024-09-14T06:10:15.648 INFO:teuthology.orchestra.run.smithi191.stderr: 77.1% -- replaced with /var/log/ceph/cluster2-client.mirror.45537.log.gz 2024-09-14T06:10:15.648 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.60508.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72761.log 2024-09-14T06:10:15.648 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.66468.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41387.log 2024-09-14T06:10:15.648 INFO:teuthology.orchestra.run.smithi191.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.60508.log.gz 2024-09-14T06:10:15.649 INFO:teuthology.orchestra.run.smithi191.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66468.log.gz 2024-09-14T06:10:15.649 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.72761.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36929.log 2024-09-14T06:10:15.649 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72761.log.gz 2024-09-14T06:10:15.649 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61703.log 2024-09-14T06:10:15.649 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.41387.log: /var/log/ceph/cluster1-client.mirror.36929.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49540.log 2024-09-14T06:10:15.649 INFO:teuthology.orchestra.run.smithi191.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.41387.log.gz 2024-09-14T06:10:15.649 INFO:teuthology.orchestra.run.smithi191.stderr: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.3.39996.log.gz 2024-09-14T06:10:15.649 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.61703.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72261.log 2024-09-14T06:10:15.649 INFO:teuthology.orchestra.run.smithi191.stderr: 87.5%/var/log/ceph/cluster2-client.mirror.49540.log: -- replaced with /var/log/ceph/cluster1-client.mirror.61703.log.gz 2024-09-14T06:10:15.649 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71933.log 2024-09-14T06:10:15.649 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.72261.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41837.log 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.72261.log.gz 2024-09-14T06:10:15.649 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-14T06:10:15.649 INFO:teuthology.orchestra.run.smithi191.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.38627.log.gz 2024-09-14T06:10:15.649 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.71933.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49540.log.gz 2024-09-14T06:10:15.650 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68742.log 2024-09-14T06:10:15.650 INFO:teuthology.orchestra.run.smithi191.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.71933.log.gz 2024-09-14T06:10:15.650 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70104.log 2024-09-14T06:10:15.650 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.41837.log: /var/log/ceph/cluster1-client.mirror.68742.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61563.log 2024-09-14T06:10:15.650 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.68742.log.gz 2024-09-14T06:10:15.650 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.70104.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57297.log 2024-09-14T06:10:15.650 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.61563.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69626.log 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.70104.log.gz 2024-09-14T06:10:15.650 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-14T06:10:15.650 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.57297.log: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41837.log.gz 2024-09-14T06:10:15.650 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58467.log 2024-09-14T06:10:15.650 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.69626.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50561.log 2024-09-14T06:10:15.650 INFO:teuthology.orchestra.run.smithi191.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69626.log.gz 2024-09-14T06:10:15.650 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.58467.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54427.log 2024-09-14T06:10:15.651 INFO:teuthology.orchestra.run.smithi191.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61563.log.gz 2024-09-14T06:10:15.651 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4%/var/log/ceph/cluster1-client.mirror.50561.log: -- replaced with /var/log/ceph/cluster1-client.mirror.58467.log.gz 2024-09-14T06:10:15.651 INFO:teuthology.orchestra.run.smithi191.stderr: 92.5%gzip -- replaced with /var/log/ceph/cluster2-client.mirror.57297.log.gz -5 2024-09-14T06:10:15.651 INFO:teuthology.orchestra.run.smithi191.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.48258.log 2024-09-14T06:10:15.651 INFO:teuthology.orchestra.run.smithi191.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.50561.log.gz 2024-09-14T06:10:15.651 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56323.log 2024-09-14T06:10:15.651 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.54427.log: /var/log/ceph/cluster2-client.mirror.48258.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66918.log 2024-09-14T06:10:15.651 INFO:teuthology.orchestra.run.smithi191.stderr: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48258.log.gz 2024-09-14T06:10:15.651 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.56323.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54186.log 2024-09-14T06:10:15.651 INFO:teuthology.orchestra.run.smithi191.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.54427.log.gz 2024-09-14T06:10:15.651 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.66918.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58009.log 2024-09-14T06:10:15.651 INFO:teuthology.orchestra.run.smithi191.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.66918.log.gz 2024-09-14T06:10:15.651 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.54186.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65976.log 2024-09-14T06:10:15.651 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.58009.log: -- /var/log/ceph/cluster2-client.mirror.61585.log 2024-09-14T06:10:15.652 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65976.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.54186.log.gz 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58009.log.gz 2024-09-14T06:10:15.652 INFO:teuthology.orchestra.run.smithi191.stderr: 80.9% -- replaced with /var/log/ceph/cluster1-client.mirror.36929.log.gz 2024-09-14T06:10:15.652 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-14T06:10:15.652 INFO:teuthology.orchestra.run.smithi191.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.36874.log.gz 2024-09-14T06:10:15.652 INFO:teuthology.orchestra.run.smithi191.stderr: 92.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39605.log 2024-09-14T06:10:15.652 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.56323.log.gz 2024-09-14T06:10:15.652 INFO:teuthology.orchestra.run.smithi191.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65976.log.gz 2024-09-14T06:10:15.652 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.61585.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40951.log 2024-09-14T06:10:15.652 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.39605.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46222.log 2024-09-14T06:10:15.652 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.40951.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61585.log.gz 2024-09-14T06:10:15.652 INFO:teuthology.orchestra.run.smithi191.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.40951.log.gz 2024-09-14T06:10:15.653 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40227.log 2024-09-14T06:10:15.653 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46222.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57204.log 2024-09-14T06:10:15.653 INFO:teuthology.orchestra.run.smithi191.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.40357.log.gz 2024-09-14T06:10:15.653 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.40227.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38539.log 2024-09-14T06:10:15.653 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.57204.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46222.log.gz 2024-09-14T06:10:15.653 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40409.log 2024-09-14T06:10:15.653 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57204.log.gz 2024-09-14T06:10:15.653 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.38539.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65105.log 2024-09-14T06:10:15.653 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- 88.2% /var/log/ceph/cluster1-client.mirror.66310.log 2024-09-14T06:10:15.653 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.40227.log.gz 2024-09-14T06:10:15.653 INFO:teuthology.orchestra.run.smithi191.stderr: 89.9%/var/log/ceph/cluster2-client.mirror.65105.log: -- replaced with /var/log/ceph/cluster2-client.mirror.39605.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42950.log 2024-09-14T06:10:15.653 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-14T06:10:15.653 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.40409.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40754.log 2024-09-14T06:10:15.654 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.66310.log: 87.7% 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66310.log.gz 2024-09-14T06:10:15.654 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.40409.log.gz 2024-09-14T06:10:15.654 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.42950.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60231.log 2024-09-14T06:10:15.654 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.40754.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47461.log 2024-09-14T06:10:15.654 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.60231.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.42950.log.gz 2024-09-14T06:10:15.654 INFO:teuthology.orchestra.run.smithi191.stderr:gzip 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.40754.log.gz 2024-09-14T06:10:15.654 INFO:teuthology.orchestra.run.smithi191.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63580.log 2024-09-14T06:10:15.654 INFO:teuthology.orchestra.run.smithi191.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60231.log.gz 2024-09-14T06:10:15.654 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64795.log 2024-09-14T06:10:15.654 INFO:teuthology.orchestra.run.smithi191.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.38583.log.gz 2024-09-14T06:10:15.654 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47461.log: /var/log/ceph/cluster2-client.mirror.63580.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65105.log.gz 2024-09-14T06:10:15.655 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39906.log 2024-09-14T06:10:15.655 INFO:teuthology.orchestra.run.smithi191.stderr: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47461.log.gz 2024-09-14T06:10:15.655 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64795.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63580.log.gz 2024-09-14T06:10:15.655 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41687.log 2024-09-14T06:10:15.655 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.39906.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70148.log 2024-09-14T06:10:15.655 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.41687.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48603.log 2024-09-14T06:10:15.655 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.70148.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56126.log 2024-09-14T06:10:15.655 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48603.log: gzip -5 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48603.log.gz 2024-09-14T06:10:15.655 INFO:teuthology.orchestra.run.smithi191.stderr: --verbose -- /var/log/ceph/cluster1-client.mirror.66513.log 2024-09-14T06:10:15.655 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72550.log 2024-09-14T06:10:15.655 INFO:teuthology.orchestra.run.smithi191.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64795.log.gz 2024-09-14T06:10:15.655 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.66513.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.70148.log.gz 2024-09-14T06:10:15.655 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.56126.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57397.log 2024-09-14T06:10:15.655 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.38539.log.gz 2024-09-14T06:10:15.656 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.72550.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72550.log.gz 2024-09-14T06:10:15.656 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66423.log 2024-09-14T06:10:15.656 INFO:teuthology.orchestra.run.smithi191.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.66513.log.gz 2024-09-14T06:10:15.656 INFO:teuthology.orchestra.run.smithi191.stderr:gzip/var/log/ceph/cluster2-client.mirror.57397.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69688.log 2024-09-14T06:10:15.656 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.66423.log: gzip -5 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56126.log.gz 2024-09-14T06:10:15.656 INFO:teuthology.orchestra.run.smithi191.stderr: --verbose 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.57397.log.gz 2024-09-14T06:10:15.656 INFO:teuthology.orchestra.run.smithi191.stderr: -- /var/log/ceph/cluster2-client.mirror.36792.log 2024-09-14T06:10:15.656 INFO:teuthology.orchestra.run.smithi191.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66423.log.gz 2024-09-14T06:10:15.656 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62829.log 2024-09-14T06:10:15.656 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.69688.log: /var/log/ceph/cluster2-client.mirror.36792.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48354.log 2024-09-14T06:10:15.656 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.62829.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67885.log 2024-09-14T06:10:15.656 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48354.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.41687.log.gz 2024-09-14T06:10:15.656 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58732.log 2024-09-14T06:10:15.657 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.67885.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45170.log 2024-09-14T06:10:15.657 INFO:teuthology.orchestra.run.smithi191.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67885.log.gz 2024-09-14T06:10:15.657 INFO:teuthology.orchestra.run.smithi191.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48354.log.gz 2024-09-14T06:10:15.657 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.58732.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.58732.log.gz 2024-09-14T06:10:15.657 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59905.log 2024-09-14T06:10:15.657 INFO:teuthology.orchestra.run.smithi191.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.62829.log.gz 2024-09-14T06:10:15.657 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45170.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71413.log 2024-09-14T06:10:15.657 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.59905.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65002.log 2024-09-14T06:10:15.657 INFO:teuthology.orchestra.run.smithi191.stderr: 91.1% 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.59905.log.gz/var/log/ceph/cluster1-client.mirror.71413.log: 2024-09-14T06:10:15.657 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72719.log 2024-09-14T06:10:15.657 INFO:teuthology.orchestra.run.smithi191.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71413.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.69688.log.gz 2024-09-14T06:10:15.657 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-14T06:10:15.657 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65002.log: gzip 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45170.log.gz 2024-09-14T06:10:15.658 INFO:teuthology.orchestra.run.smithi191.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51639.log 2024-09-14T06:10:15.658 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.72719.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72719.log.gz 2024-09-14T06:10:15.658 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65002.log.gz 2024-09-14T06:10:15.658 INFO:teuthology.orchestra.run.smithi191.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.2.44429.log 2024-09-14T06:10:15.658 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.51639.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58337.log 2024-09-14T06:10:15.658 INFO:teuthology.orchestra.run.smithi191.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.51639.log.gz 2024-09-14T06:10:15.658 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.2.44429.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64526.log 2024-09-14T06:10:15.658 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.58337.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63413.log 2024-09-14T06:10:15.658 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64526.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53817.log 2024-09-14T06:10:15.658 INFO:teuthology.orchestra.run.smithi191.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58337.log.gz 2024-09-14T06:10:15.658 INFO:teuthology.orchestra.run.smithi191.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64526.log.gz 2024-09-14T06:10:15.658 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.63413.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42170.log 2024-09-14T06:10:15.658 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.53817.log: -- /var/log/ceph/cluster1-client.admin.37225.log 2024-09-14T06:10:15.659 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.42170.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53817.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44638.log 2024-09-14T06:10:15.659 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-14T06:10:15.659 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50264.log 2024-09-14T06:10:15.659 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.37225.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37225.log.gz 2024-09-14T06:10:15.659 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.44638.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40155.log 2024-09-14T06:10:15.659 INFO:teuthology.orchestra.run.smithi191.stderr: 90.4%/var/log/ceph/cluster1-client.mirror.50264.log: -- replaced with /var/log/ceph/cluster2-client.mirror.63413.log.gz 2024-09-14T06:10:15.659 INFO:teuthology.orchestra.run.smithi191.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44638.log.gz 2024-09-14T06:10:15.659 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.50264.log.gz 2024-09-14T06:10:15.659 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46968.log 2024-09-14T06:10:15.659 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.40155.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40111.log 2024-09-14T06:10:15.659 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54123.log 2024-09-14T06:10:15.659 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.40111.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.40155.log.gz 2024-09-14T06:10:15.660 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46968.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52266.log 2024-09-14T06:10:15.660 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.54123.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46968.log.gz 2024-09-14T06:10:15.660 INFO:teuthology.orchestra.run.smithi191.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.54123.log.gz 2024-09-14T06:10:15.660 INFO:teuthology.orchestra.run.smithi191.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.40111.log.gz 2024-09-14T06:10:15.660 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46401.log 2024-09-14T06:10:15.660 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.52266.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39303.log 2024-09-14T06:10:15.660 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52266.log.gz 2024-09-14T06:10:15.660 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46401.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53439.log 2024-09-14T06:10:15.660 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46401.log.gz 2024-09-14T06:10:15.660 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.39303.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39809.log 2024-09-14T06:10:15.660 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.53439.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49582.log 2024-09-14T06:10:15.660 INFO:teuthology.orchestra.run.smithi191.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42170.log.gz 2024-09-14T06:10:15.661 INFO:teuthology.orchestra.run.smithi191.stderr:gzip 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.53439.log.gz 2024-09-14T06:10:15.661 INFO:teuthology.orchestra.run.smithi191.stderr: -5 --verbose/var/log/ceph/cluster2-client.mirror.39809.log: -- /var/log/ceph/cluster1-client.admin.37201.log 2024-09-14T06:10:15.661 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68784.log 2024-09-14T06:10:15.661 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.37201.log: /var/log/ceph/cluster2-client.mirror.49582.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50714.log 2024-09-14T06:10:15.661 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.37201.log.gz 2024-09-14T06:10:15.661 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.68784.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37607.log 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68784.log.gz 2024-09-14T06:10:15.661 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-14T06:10:15.661 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.50714.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55772.log 2024-09-14T06:10:15.661 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.50714.log.gz 2024-09-14T06:10:15.661 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.37607.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71139.log 2024-09-14T06:10:15.661 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.55772.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55772.log.gz 2024-09-14T06:10:15.661 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.71139.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.71139.log.gz 2024-09-14T06:10:15.661 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51318.log 2024-09-14T06:10:15.661 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36886.log 2024-09-14T06:10:15.662 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.51318.log: 53.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41247.log 2024-09-14T06:10:15.662 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.37607.log.gz 2024-09-14T06:10:15.662 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.51318.log.gz 2024-09-14T06:10:15.662 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.36886.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44240.log 2024-09-14T06:10:15.662 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.41247.log: 91.5% -- replaced with /var/log/ceph/cluster1-client.mirror.36886.log.gz 2024-09-14T06:10:15.662 INFO:teuthology.orchestra.run.smithi191.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.0.36874.log.gz 2024-09-14T06:10:15.662 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51011.log 2024-09-14T06:10:15.662 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.44240.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.41247.log.gz 2024-09-14T06:10:15.662 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51840.log 2024-09-14T06:10:15.662 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.51011.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64671.log 2024-09-14T06:10:15.662 INFO:teuthology.orchestra.run.smithi191.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.51011.log.gz 2024-09-14T06:10:15.663 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.51840.log: gzip -5 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44240.log.gz 2024-09-14T06:10:15.663 INFO:teuthology.orchestra.run.smithi191.stderr: --verbose -- /var/log/ceph/cluster1-client.mirror.45448.log 2024-09-14T06:10:15.663 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64671.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.51840.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63724.log 2024-09-14T06:10:15.663 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-14T06:10:15.663 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.45448.log: gzip 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.40315.log.gz 2024-09-14T06:10:15.663 INFO:teuthology.orchestra.run.smithi191.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60990.log 2024-09-14T06:10:15.678 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.63724.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63724.log.gz 2024-09-14T06:10:15.678 INFO:teuthology.orchestra.run.smithi191.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.39809.log.gz 2024-09-14T06:10:15.678 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69952.log 2024-09-14T06:10:15.678 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.60990.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64671.log.gz 2024-09-14T06:10:15.678 INFO:teuthology.orchestra.run.smithi191.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.60990.log.gz 2024-09-14T06:10:15.679 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49560.log 2024-09-14T06:10:15.679 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.69952.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.56874.log 2024-09-14T06:10:15.679 INFO:teuthology.orchestra.run.smithi191.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69952.log.gz 2024-09-14T06:10:15.679 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49560.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49560.log.gz 2024-09-14T06:10:15.679 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69516.log 2024-09-14T06:10:15.679 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.1.56874.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.39303.log.gz 2024-09-14T06:10:15.680 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46242.log 2024-09-14T06:10:15.680 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.69516.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.39906.log.gz 2024-09-14T06:10:15.680 INFO:teuthology.orchestra.run.smithi191.stderr: 95.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45448.log.gz 2024-09-14T06:10:15.680 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68065.log 2024-09-14T06:10:15.680 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46242.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46242.log.gz 2024-09-14T06:10:15.681 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58054.log 2024-09-14T06:10:15.681 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.68065.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68065.log.gz 2024-09-14T06:10:15.681 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46264.log 2024-09-14T06:10:15.681 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.58054.log: 93.6% -- replaced with /var/log/ceph/cluster1-client.mirror.1.56874.log.gz 2024-09-14T06:10:15.681 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48719.log 2024-09-14T06:10:15.681 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46264.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58054.log.gz 2024-09-14T06:10:15.682 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43283.log 2024-09-14T06:10:15.682 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48719.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48719.log.gz 2024-09-14T06:10:15.682 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40335.log 2024-09-14T06:10:15.682 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.43283.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43283.log.gz 2024-09-14T06:10:15.682 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46661.log 2024-09-14T06:10:15.682 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.40335.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67863.log 2024-09-14T06:10:15.683 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46661.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.40335.log.gz 2024-09-14T06:10:15.683 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58202.log 2024-09-14T06:10:15.683 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.67863.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67863.log.gz 2024-09-14T06:10:15.684 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57276.log 2024-09-14T06:10:15.695 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.58202.log: gzip -5 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58202.log.gz 2024-09-14T06:10:15.695 INFO:teuthology.orchestra.run.smithi191.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.48821.log 2024-09-14T06:10:15.695 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.57276.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.57276.log.gz 2024-09-14T06:10:15.706 INFO:teuthology.orchestra.run.smithi191.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.2.44429.log.gz 2024-09-14T06:10:15.706 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58592.log 2024-09-14T06:10:15.723 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48821.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42387.log 2024-09-14T06:10:15.724 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.58592.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.58592.log.gz 2024-09-14T06:10:15.736 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52668.log 2024-09-14T06:10:15.737 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.42387.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.42387.log.gz 2024-09-14T06:10:15.758 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39479.log 2024-09-14T06:10:15.759 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.52668.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52668.log.gz 2024-09-14T06:10:15.771 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45976.log 2024-09-14T06:10:15.785 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.39479.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67503.log 2024-09-14T06:10:15.785 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45976.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45976.log.gz 2024-09-14T06:10:15.793 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58033.log 2024-09-14T06:10:15.806 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.67503.log: 86.5% -- replaced with /var/log/ceph/cluster1-client.mirror.67503.log.gz 2024-09-14T06:10:15.807 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51748.log 2024-09-14T06:10:15.808 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.58033.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.58033.log.gz 2024-09-14T06:10:15.820 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46560.log 2024-09-14T06:10:15.821 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.51748.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51748.log.gz 2024-09-14T06:10:15.829 INFO:teuthology.orchestra.run.smithi143.stderr: 91.3% -- replaced with /var/log/ceph/cluster1-mgr.x.log.gz 2024-09-14T06:10:15.833 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.63179.log 2024-09-14T06:10:15.833 INFO:teuthology.orchestra.run.smithi143.stderr: 91.2% -- replaced with /var/log/ceph/cluster2-mgr.x.log.gz 2024-09-14T06:10:15.834 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46560.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46560.log.gz 2024-09-14T06:10:15.845 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54385.log 2024-09-14T06:10:15.854 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.2.63179.log: 93.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65374.log 2024-09-14T06:10:15.863 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.54385.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67795.log 2024-09-14T06:10:15.864 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65374.log: -- replaced with /var/log/ceph/cluster2-client.mirror.2.63179.log.gz 2024-09-14T06:10:15.868 INFO:teuthology.orchestra.run.smithi191.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54385.log.gz 2024-09-14T06:10:15.877 INFO:teuthology.orchestra.run.smithi191.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65374.log.gz 2024-09-14T06:10:15.887 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62534.log 2024-09-14T06:10:15.887 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.67795.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67795.log.gz 2024-09-14T06:10:15.901 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38605.log 2024-09-14T06:10:15.902 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.62534.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.62534.log.gz 2024-09-14T06:10:15.905 INFO:teuthology.orchestra.run.smithi191.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.68949.log.gz 2024-09-14T06:10:15.906 INFO:teuthology.orchestra.run.smithi191.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49582.log.gz 2024-09-14T06:10:15.915 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43982.log 2024-09-14T06:10:15.928 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.38605.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42838.log 2024-09-14T06:10:15.928 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.38605.log.gz 2024-09-14T06:10:15.941 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.43982.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32367.log 2024-09-14T06:10:15.942 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.42838.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.42838.log.gz 2024-09-14T06:10:15.946 INFO:teuthology.orchestra.run.smithi191.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43982.log.gz 2024-09-14T06:10:15.954 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37471.log 2024-09-14T06:10:15.954 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.32367.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32367.log.gz 2024-09-14T06:10:15.964 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65271.log 2024-09-14T06:10:15.973 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.37471.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58491.log 2024-09-14T06:10:15.976 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65271.log: 88.5% 80.3% -- replaced with /var/log/ceph/cluster2-client.admin.37471.log.gz 2024-09-14T06:10:15.981 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65147.log 2024-09-14T06:10:15.986 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.58491.log: -- replaced with /var/log/ceph/cluster2-client.mirror.65271.log.gz 2024-09-14T06:10:15.993 INFO:teuthology.orchestra.run.smithi191.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.70052.log.gz 2024-09-14T06:10:15.993 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43853.log 2024-09-14T06:10:15.994 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65147.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65147.log.gz 2024-09-14T06:10:15.995 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51274.log 2024-09-14T06:10:15.995 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.43853.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.58491.log.gz 2024-09-14T06:10:16.001 INFO:teuthology.orchestra.run.smithi191.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43853.log.gz 2024-09-14T06:10:16.001 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55053.log 2024-09-14T06:10:16.002 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.51274.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46264.log.gz 2024-09-14T06:10:16.002 INFO:teuthology.orchestra.run.smithi191.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51274.log.gz 2024-09-14T06:10:16.003 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39766.log 2024-09-14T06:10:16.003 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.55053.log: -- /var/log/ceph/cluster2-client.mirror.58833.log 2024-09-14T06:10:16.006 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.39766.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62871.log 2024-09-14T06:10:16.006 INFO:teuthology.orchestra.run.smithi191.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.55053.log.gz 2024-09-14T06:10:16.006 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.58833.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47498.log 2024-09-14T06:10:16.006 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.62871.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62871.log.gz 2024-09-14T06:10:16.006 INFO:teuthology.orchestra.run.smithi191.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58833.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51702.log 2024-09-14T06:10:16.006 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-14T06:10:16.006 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47498.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47498.log.gz 2024-09-14T06:10:16.011 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- 94.2% -- replaced with /var/log/ceph/cluster2-client.mirror.36792.log.gz 2024-09-14T06:10:16.011 INFO:teuthology.orchestra.run.smithi191.stderr: /var/log/ceph/cluster1-client.mirror.39697.log 2024-09-14T06:10:16.011 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.51702.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65167.log 2024-09-14T06:10:16.012 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.39697.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63497.log 2024-09-14T06:10:16.012 INFO:teuthology.orchestra.run.smithi191.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51702.log.gz 2024-09-14T06:10:16.012 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65167.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.39697.log.gz 2024-09-14T06:10:16.012 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62438.log 2024-09-14T06:10:16.012 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.63497.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48476.log 2024-09-14T06:10:16.012 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.62438.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.63497.log.gz 2024-09-14T06:10:16.012 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62438.log.gz 2024-09-14T06:10:16.012 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54626.log 2024-09-14T06:10:16.012 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.48476.log: -- /var/log/ceph/cluster1-client.mirror.52989.log 2024-09-14T06:10:16.012 INFO:teuthology.orchestra.run.smithi191.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65167.log.gz 2024-09-14T06:10:16.012 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.54626.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56712.log 2024-09-14T06:10:16.012 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.52989.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.52989.log.gz 2024-09-14T06:10:16.017 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44778.log 2024-09-14T06:10:16.017 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.56712.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56712.log.gz 2024-09-14T06:10:16.017 INFO:teuthology.orchestra.run.smithi191.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.54626.log.gz 2024-09-14T06:10:16.017 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48116.log 2024-09-14T06:10:16.017 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.44778.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55889.log 2024-09-14T06:10:16.017 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48116.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44778.log.gz 2024-09-14T06:10:16.018 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62912.log 2024-09-14T06:10:16.018 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.55889.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48116.log.gz 2024-09-14T06:10:16.018 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40462.log 2024-09-14T06:10:16.018 INFO:teuthology.orchestra.run.smithi191.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.55889.log.gz 2024-09-14T06:10:16.018 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.62912.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62912.log.gz 2024-09-14T06:10:16.018 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67975.log 2024-09-14T06:10:16.114 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.40462.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41811.log 2024-09-14T06:10:16.114 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.67975.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.59529.log 2024-09-14T06:10:16.114 INFO:teuthology.orchestra.run.smithi191.stderr: 87.0%/var/log/ceph/cluster1-client.admin.41811.log: -- replaced with /var/log/ceph/cluster1-client.mirror.67975.log.gz 2024-09-14T06:10:16.114 INFO:teuthology.orchestra.run.smithi191.stderr: 87.8%gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32744.log 2024-09-14T06:10:16.114 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.41811.log.gz 2024-09-14T06:10:16.114 INFO:teuthology.orchestra.run.smithi191.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.68431.log.gz 2024-09-14T06:10:16.114 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.0.59529.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67030.log 2024-09-14T06:10:16.114 INFO:teuthology.orchestra.run.smithi191.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.40462.log.gz 2024-09-14T06:10:16.114 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.32744.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64220.log 2024-09-14T06:10:16.115 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32744.log.gz 2024-09-14T06:10:16.115 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5/var/log/ceph/cluster1-client.mirror.67030.log: --verbose -- /var/log/ceph/cluster1-client.mirror.72021.log 2024-09-14T06:10:16.115 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64220.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67030.log.gz 2024-09-14T06:10:16.115 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63848.log 2024-09-14T06:10:16.115 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.72021.log: gzip -5 --verbose 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72021.log.gz 2024-09-14T06:10:16.115 INFO:teuthology.orchestra.run.smithi191.stderr: -- /var/log/ceph/cluster2-client.mirror.60641.log 2024-09-14T06:10:16.115 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.63848.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40861.log 2024-09-14T06:10:16.115 INFO:teuthology.orchestra.run.smithi191.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.0.59529.log.gz 2024-09-14T06:10:16.115 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.60641.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35943.log 2024-09-14T06:10:16.115 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.40861.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60641.log.gz 2024-09-14T06:10:16.115 INFO:teuthology.orchestra.run.smithi191.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64220.log.gz 2024-09-14T06:10:16.115 INFO:teuthology.orchestra.run.smithi191.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.40861.log.gz 2024-09-14T06:10:16.115 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46540.log 2024-09-14T06:10:16.116 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.admin.35943.log: -- /var/log/ceph/cluster2-client.mirror.46523.log 2024-09-14T06:10:16.116 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35943.log.gz 2024-09-14T06:10:16.116 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46540.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46126.log 2024-09-14T06:10:16.116 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46523.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46523.log.gz 2024-09-14T06:10:16.116 INFO:teuthology.orchestra.run.smithi191.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46540.log.gz 2024-09-14T06:10:16.116 INFO:teuthology.orchestra.run.smithi191.stderr: 90.3%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44369.log 2024-09-14T06:10:16.116 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.63848.log.gz 2024-09-14T06:10:16.116 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46126.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50312.log 2024-09-14T06:10:16.116 INFO:teuthology.orchestra.run.smithi191.stderr: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46126.log.gz 2024-09-14T06:10:16.116 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.44369.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54774.log 2024-09-14T06:10:16.116 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5/var/log/ceph/cluster1-client.mirror.50312.log: --verbose -- /var/log/ceph/cluster2-client.mirror.39266.log 2024-09-14T06:10:16.116 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.54774.log: gzip 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.50312.log.gz 2024-09-14T06:10:16.116 INFO:teuthology.orchestra.run.smithi191.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41579.log 2024-09-14T06:10:16.116 INFO:teuthology.orchestra.run.smithi191.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.54774.log.gz 2024-09-14T06:10:16.117 INFO:teuthology.orchestra.run.smithi191.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.69516.log.gz 2024-09-14T06:10:16.117 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37334.log/var/log/ceph/cluster2-client.mirror.39266.log: 2024-09-14T06:10:16.117 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.41579.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52355.log 2024-09-14T06:10:16.117 INFO:teuthology.orchestra.run.smithi191.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41579.log.gz 2024-09-14T06:10:16.117 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.37334.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55234.log 2024-09-14T06:10:16.117 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45574.log 2024-09-14T06:10:16.117 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.52355.log: /var/log/ceph/cluster1-client.mirror.55234.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67413.log 2024-09-14T06:10:16.117 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45574.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71033.log 2024-09-14T06:10:16.117 INFO:teuthology.orchestra.run.smithi191.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45574.log.gz 2024-09-14T06:10:16.117 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.67413.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67413.log.gz 2024-09-14T06:10:16.117 INFO:teuthology.orchestra.run.smithi191.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52355.log.gz 2024-09-14T06:10:16.117 INFO:teuthology.orchestra.run.smithi191.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55234.log.gz 2024-09-14T06:10:16.117 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61171.log 2024-09-14T06:10:16.118 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.71033.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37402.log 2024-09-14T06:10:16.118 INFO:teuthology.orchestra.run.smithi191.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.71033.log.gz 2024-09-14T06:10:16.118 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.61171.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44658.log 2024-09-14T06:10:16.118 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.37402.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.61171.log.gz 2024-09-14T06:10:16.118 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66760.log 2024-09-14T06:10:16.118 INFO:teuthology.orchestra.run.smithi191.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44369.log.gz 2024-09-14T06:10:16.118 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71077.log 2024-09-14T06:10:16.118 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.44658.log: /var/log/ceph/cluster1-client.mirror.66760.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61195.log 2024-09-14T06:10:16.118 INFO:teuthology.orchestra.run.smithi191.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66760.log.gz 2024-09-14T06:10:16.118 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.71077.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44658.log.gz 2024-09-14T06:10:16.118 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48158.log 2024-09-14T06:10:16.118 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- 88.2% /var/log/ceph/cluster2-client.admin.37505.log 2024-09-14T06:10:16.118 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.71077.log.gz 2024-09-14T06:10:16.119 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.61195.log: /var/log/ceph/cluster2-client.mirror.48158.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48099.log 2024-09-14T06:10:16.119 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61195.log.gz 2024-09-14T06:10:16.119 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.37505.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60114.log 2024-09-14T06:10:16.119 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 3.5% -- replaced with /var/log/ceph/cluster1-client.mirror.37334.log.gz 2024-09-14T06:10:16.119 INFO:teuthology.orchestra.run.smithi191.stderr: --verbose/var/log/ceph/cluster2-client.mirror.48099.log: -- /var/log/ceph/cluster2-client.mirror.62705.log 2024-09-14T06:10:16.119 INFO:teuthology.orchestra.run.smithi191.stderr: 80.4% 54.5% -- replaced with /var/log/ceph/cluster1-client.mirror.37402.log.gz 2024-09-14T06:10:16.119 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.48099.log.gz 2024-09-14T06:10:16.119 INFO:teuthology.orchestra.run.smithi191.stderr: 66.7% -- replaced with /var/log/ceph/cluster2-client.admin.37505.log.gz 2024-09-14T06:10:16.119 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.60114.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64076.log 2024-09-14T06:10:16.119 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.62705.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50084.log 2024-09-14T06:10:16.119 INFO:teuthology.orchestra.run.smithi191.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60114.log.gz 2024-09-14T06:10:16.119 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64076.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72042.log 2024-09-14T06:10:16.119 INFO:teuthology.orchestra.run.smithi191.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.39766.log.gz 2024-09-14T06:10:16.120 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.50084.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64076.log.gz 2024-09-14T06:10:16.120 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43462.log 2024-09-14T06:10:16.120 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.72042.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40600.log 2024-09-14T06:10:16.120 INFO:teuthology.orchestra.run.smithi191.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.72042.log.gz 2024-09-14T06:10:16.120 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.43462.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.62705.log.gz 2024-09-14T06:10:16.120 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38907.log 2024-09-14T06:10:16.120 INFO:teuthology.orchestra.run.smithi191.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43462.log.gz 2024-09-14T06:10:16.120 INFO:teuthology.orchestra.run.smithi191.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50084.log.gz 2024-09-14T06:10:16.120 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.40600.log: -- /var/log/ceph/cluster2-client.mirror.63518.log 2024-09-14T06:10:16.120 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.38907.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.40600.log.gz 2024-09-14T06:10:16.120 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52130.log 2024-09-14T06:10:16.120 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.63518.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63456.log 2024-09-14T06:10:16.121 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.52130.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63518.log.gz 2024-09-14T06:10:16.121 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42299.log 2024-09-14T06:10:16.121 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.63456.log: gzip 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63456.log.gz/var/log/ceph/cluster2-client.mirror.42299.log: 2024-09-14T06:10:16.121 INFO:teuthology.orchestra.run.smithi191.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52130.log.gz 2024-09-14T06:10:16.121 INFO:teuthology.orchestra.run.smithi191.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62995.log 2024-09-14T06:10:16.121 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57160.log 2024-09-14T06:10:16.121 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.62995.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59256.log 2024-09-14T06:10:16.121 INFO:teuthology.orchestra.run.smithi191.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.62995.log.gz 2024-09-14T06:10:16.121 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.57160.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57160.log.gz 2024-09-14T06:10:16.121 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67998.log 2024-09-14T06:10:16.121 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58660.log 2024-09-14T06:10:16.121 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.67998.log: /var/log/ceph/cluster1-client.mirror.59256.log: gzip -5 --verbose 87.0% -- /var/log/ceph/cluster1-client.mirror.51792.log 2024-09-14T06:10:16.121 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.67998.log.gz 2024-09-14T06:10:16.121 INFO:teuthology.orchestra.run.smithi191.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42299.log.gz 2024-09-14T06:10:16.122 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.58660.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60874.log 2024-09-14T06:10:16.122 INFO:teuthology.orchestra.run.smithi191.stderr: 92.3% 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.58660.log.gz 2024-09-14T06:10:16.122 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.59256.log.gz 2024-09-14T06:10:16.122 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.51792.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52602.log 2024-09-14T06:10:16.122 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.60874.log: 93.4% 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.51792.log.gz 2024-09-14T06:10:16.122 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.39266.log.gz 2024-09-14T06:10:16.122 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38929.log 2024-09-14T06:10:16.122 INFO:teuthology.orchestra.run.smithi191.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.60874.log.gz/var/log/ceph/cluster2-client.mirror.52602.log: 2024-09-14T06:10:16.122 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.42207.log 2024-09-14T06:10:16.122 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.38929.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64241.log 2024-09-14T06:10:16.122 INFO:teuthology.orchestra.run.smithi191.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.52602.log.gz 2024-09-14T06:10:16.122 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.3.42207.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46037.log 2024-09-14T06:10:16.123 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64241.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53277.log 2024-09-14T06:10:16.123 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.46037.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41158.log 2024-09-14T06:10:16.123 INFO:teuthology.orchestra.run.smithi191.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.46037.log.gz 2024-09-14T06:10:16.123 INFO:teuthology.orchestra.run.smithi191.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64241.log.gz 2024-09-14T06:10:16.123 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.53277.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44698.log 2024-09-14T06:10:16.123 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.41158.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58636.log 2024-09-14T06:10:16.123 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.44698.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.41158.log.gz 2024-09-14T06:10:16.123 INFO:teuthology.orchestra.run.smithi191.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53277.log.gz 2024-09-14T06:10:16.123 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60571.log 2024-09-14T06:10:16.123 INFO:teuthology.orchestra.run.smithi191.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44698.log.gz 2024-09-14T06:10:16.123 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.58636.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41462.log 2024-09-14T06:10:16.123 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58636.log.gz 2024-09-14T06:10:16.123 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49812.log 2024-09-14T06:10:16.123 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.60571.log: /var/log/ceph/cluster2-client.mirror.41462.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53010.log 2024-09-14T06:10:16.123 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49812.log: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.49812.log.gz 2024-09-14T06:10:16.124 INFO:teuthology.orchestra.run.smithi191.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.60571.log.gz 2024-09-14T06:10:16.124 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.41462.log.gz 2024-09-14T06:10:16.124 INFO:teuthology.orchestra.run.smithi191.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.3.42207.log.gz 2024-09-14T06:10:16.124 INFO:teuthology.orchestra.run.smithi191.stderr: -- /var/log/ceph/cluster1-client.mirror.62185.log 2024-09-14T06:10:16.124 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.53010.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47977.log 2024-09-14T06:10:16.124 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.62185.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55145.log 2024-09-14T06:10:16.124 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62185.log.gz 2024-09-14T06:10:16.124 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47977.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55314.log 2024-09-14T06:10:16.124 INFO:teuthology.orchestra.run.smithi191.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47977.log.gz 2024-09-14T06:10:16.124 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.55145.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52065.log 2024-09-14T06:10:16.124 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55145.log.gz 2024-09-14T06:10:16.124 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50786.log 2024-09-14T06:10:16.124 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.55314.log: /var/log/ceph/cluster1-client.mirror.52065.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65645.log 2024-09-14T06:10:16.124 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55314.log.gz 2024-09-14T06:10:16.125 INFO:teuthology.orchestra.run.smithi191.stderr: 87.7%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39245.log 2024-09-14T06:10:16.125 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.52065.log.gz 2024-09-14T06:10:16.125 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.50786.log: /var/log/ceph/cluster2-client.mirror.65645.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41624.log 2024-09-14T06:10:16.125 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.50786.log.gz 2024-09-14T06:10:16.125 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.39245.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70692.log 2024-09-14T06:10:16.125 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.41624.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67930.log 2024-09-14T06:10:16.125 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.70692.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48216.log 2024-09-14T06:10:16.125 INFO:teuthology.orchestra.run.smithi191.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.70692.log.gz 2024-09-14T06:10:16.125 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.67930.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67930.log.gz 2024-09-14T06:10:16.125 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65645.log.gz 2024-09-14T06:10:16.125 INFO:teuthology.orchestra.run.smithi191.stderr: -- /var/log/ceph/cluster2-client.mirror.65790.log 2024-09-14T06:10:16.125 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48216.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48057.log 2024-09-14T06:10:16.125 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65790.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.39245.log.gz 2024-09-14T06:10:16.125 INFO:teuthology.orchestra.run.smithi191.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48216.log.gz 2024-09-14T06:10:16.126 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69474.log 2024-09-14T06:10:16.126 INFO:teuthology.orchestra.run.smithi191.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65790.log.gz 2024-09-14T06:10:16.126 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48057.log: gzip 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48057.log.gz 2024-09-14T06:10:16.126 INFO:teuthology.orchestra.run.smithi191.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62091.log 2024-09-14T06:10:16.126 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.69474.log: -- /var/log/ceph/cluster1-client.mirror.38818.log 2024-09-14T06:10:16.126 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.62091.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66625.log 2024-09-14T06:10:16.126 INFO:teuthology.orchestra.run.smithi191.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.69474.log.gz 2024-09-14T06:10:16.126 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.38818.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41624.log.gz 2024-09-14T06:10:16.126 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51523.log 2024-09-14T06:10:16.126 INFO:teuthology.orchestra.run.smithi191.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62091.log.gz 2024-09-14T06:10:16.126 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.66625.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64567.log 2024-09-14T06:10:16.126 INFO:teuthology.orchestra.run.smithi191.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66625.log.gz 2024-09-14T06:10:16.126 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.51523.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60388.log 2024-09-14T06:10:16.127 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64567.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58950.log 2024-09-14T06:10:16.127 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.60388.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51523.log.gz 2024-09-14T06:10:16.127 INFO:teuthology.orchestra.run.smithi191.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64567.log.gz 2024-09-14T06:10:16.127 INFO:teuthology.orchestra.run.smithi191.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.60388.log.gz 2024-09-14T06:10:16.127 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38951.log 2024-09-14T06:10:16.127 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.58950.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45874.log 2024-09-14T06:10:16.127 INFO:teuthology.orchestra.run.smithi191.stderr: 87.7%/var/log/ceph/cluster2-client.mirror.38951.log: -- replaced with /var/log/ceph/cluster1-client.mirror.58950.log.gz 2024-09-14T06:10:16.127 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72136.log 2024-09-14T06:10:16.127 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45874.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61147.log 2024-09-14T06:10:16.127 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.72136.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72136.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67368.log 2024-09-14T06:10:16.127 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-14T06:10:16.127 INFO:teuthology.orchestra.run.smithi191.stderr:gzip 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45874.log.gz 2024-09-14T06:10:16.127 INFO:teuthology.orchestra.run.smithi191.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42129.log 2024-09-14T06:10:16.127 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.61147.log: /var/log/ceph/cluster1-client.mirror.67368.log: gzip -5 --verbose 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67368.log.gz 2024-09-14T06:10:16.128 INFO:teuthology.orchestra.run.smithi191.stderr: -- /var/log/ceph/cluster1-client.mirror.39173.log 2024-09-14T06:10:16.128 INFO:teuthology.orchestra.run.smithi191.stderr: 93.3% 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61147.log.gz 2024-09-14T06:10:16.128 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.38907.log.gz 2024-09-14T06:10:16.128 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50240.log 2024-09-14T06:10:16.128 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.42129.log: /var/log/ceph/cluster1-client.mirror.39173.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39059.log 2024-09-14T06:10:16.128 INFO:teuthology.orchestra.run.smithi191.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.42129.log.gz 2024-09-14T06:10:16.128 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.50240.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38842.log 2024-09-14T06:10:16.128 INFO:teuthology.orchestra.run.smithi191.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.50240.log.gz 2024-09-14T06:10:16.128 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.39059.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54224.log 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53010.log.gz 2024-09-14T06:10:16.128 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-14T06:10:16.128 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.38842.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53030.log 2024-09-14T06:10:16.128 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.54224.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49854.log 2024-09-14T06:10:16.128 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.53030.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70713.log 2024-09-14T06:10:16.129 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66198.log 2024-09-14T06:10:16.129 INFO:teuthology.orchestra.run.smithi191.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54224.log.gz 2024-09-14T06:10:16.129 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.70713.log: /var/log/ceph/cluster1-client.mirror.49854.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.49854.log.gz 2024-09-14T06:10:16.129 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- 88.4% /var/log/ceph/cluster2-client.mirror.62933.log 2024-09-14T06:10:16.129 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.70713.log.gz 2024-09-14T06:10:16.129 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.66198.log: gzip -5 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66198.log.gz 2024-09-14T06:10:16.129 INFO:teuthology.orchestra.run.smithi191.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.49621.log 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53030.log.gz 2024-09-14T06:10:16.129 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-14T06:10:16.129 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.62933.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44111.log 2024-09-14T06:10:16.129 INFO:teuthology.orchestra.run.smithi191.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62933.log.gz 2024-09-14T06:10:16.129 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49621.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.38818.log.gz 2024-09-14T06:10:16.129 INFO:teuthology.orchestra.run.smithi191.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.38842.log.gz/var/log/ceph/cluster1-client.mirror.44111.log: 2024-09-14T06:10:16.129 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59278.log 2024-09-14T06:10:16.130 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71977.log 2024-09-14T06:10:16.130 INFO:teuthology.orchestra.run.smithi191.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.38929.log.gz 2024-09-14T06:10:16.130 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60802.log 2024-09-14T06:10:16.130 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.59278.log: /var/log/ceph/cluster2-client.mirror.71977.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49621.log.gz 2024-09-14T06:10:16.130 INFO:teuthology.orchestra.run.smithi191.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59278.log.gz 2024-09-14T06:10:16.130 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57011.log 2024-09-14T06:10:16.130 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64816.log 2024-09-14T06:10:16.130 INFO:teuthology.orchestra.run.smithi191.stderr: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.71977.log.gz 2024-09-14T06:10:16.130 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.60802.log: /var/log/ceph/cluster1-client.mirror.57011.log: gzip 87.4% -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38770.log 2024-09-14T06:10:16.130 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.57011.log.gz 2024-09-14T06:10:16.130 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64816.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37249.log 2024-09-14T06:10:16.130 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.38770.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64816.log.gz 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.39173.log.gz 2024-09-14T06:10:16.130 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47520.log 2024-09-14T06:10:16.131 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-14T06:10:16.131 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.38770.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66445.log 2024-09-14T06:10:16.131 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-14T06:10:16.131 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47520.log: /var/log/ceph/cluster1-client.admin.37249.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.60802.log.gz 2024-09-14T06:10:16.131 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47760.log 2024-09-14T06:10:16.131 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.66445.log: 80.1% -- replaced with /var/log/ceph/cluster1-client.admin.37249.log.gz 2024-09-14T06:10:16.131 INFO:teuthology.orchestra.run.smithi191.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66445.log.gz 2024-09-14T06:10:16.131 INFO:teuthology.orchestra.run.smithi191.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44111.log.gz 2024-09-14T06:10:16.131 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72635.log 2024-09-14T06:10:16.131 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62069.log 2024-09-14T06:10:16.131 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47760.log: /var/log/ceph/cluster2-client.admin.72635.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57937.log 2024-09-14T06:10:16.131 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72635.log.gz 2024-09-14T06:10:16.131 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.62069.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68890.log 2024-09-14T06:10:16.131 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.57937.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.57937.log.gz 2024-09-14T06:10:16.131 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49396.log 2024-09-14T06:10:16.132 INFO:teuthology.orchestra.run.smithi191.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.62069.log.gz/var/log/ceph/cluster2-client.mirror.68890.log: 2024-09-14T06:10:16.132 INFO:teuthology.orchestra.run.smithi191.stderr: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.68890.log.gz 2024-09-14T06:10:16.132 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49396.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54557.log 2024-09-14T06:10:16.132 INFO:teuthology.orchestra.run.smithi191.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49396.log.gz 2024-09-14T06:10:16.132 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59015.log 2024-09-14T06:10:16.132 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.39059.log.gz 2024-09-14T06:10:16.132 INFO:teuthology.orchestra.run.smithi191.stderr: --/var/log/ceph/cluster1-client.mirror.54557.log: /var/log/ceph/cluster1-client.mirror.37317.log 2024-09-14T06:10:16.132 INFO:teuthology.orchestra.run.smithi191.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54557.log.gz 2024-09-14T06:10:16.132 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.59015.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42554.log 2024-09-14T06:10:16.132 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.37317.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64981.log 2024-09-14T06:10:16.132 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.42554.log: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.42554.log.gz 2024-09-14T06:10:16.132 INFO:teuthology.orchestra.run.smithi191.stderr: 92.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60368.log 2024-09-14T06:10:16.132 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.59015.log.gz 2024-09-14T06:10:16.133 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64981.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65666.log 2024-09-14T06:10:16.133 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.60368.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.60368.log.gz 2024-09-14T06:10:16.133 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38561.log 2024-09-14T06:10:16.133 INFO:teuthology.orchestra.run.smithi191.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64981.log.gz 2024-09-14T06:10:16.133 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67638.log 2024-09-14T06:10:16.133 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.38561.log: /var/log/ceph/cluster2-client.mirror.65666.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39435.log 2024-09-14T06:10:16.133 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.67638.log: 86.9% 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65666.log.gz 2024-09-14T06:10:16.133 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.67638.log.gz 2024-09-14T06:10:16.133 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45778.log 2024-09-14T06:10:16.133 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60436.log 2024-09-14T06:10:16.133 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45778.log: /var/log/ceph/cluster1-client.mirror.39435.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45778.log.gz 2024-09-14T06:10:16.133 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72929.log 2024-09-14T06:10:16.133 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46685.log 2024-09-14T06:10:16.133 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.72929.log: /var/log/ceph/cluster1-client.mirror.60436.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72929.log.gz 2024-09-14T06:10:16.134 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60436.log.gz 2024-09-14T06:10:16.134 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37683.log 2024-09-14T06:10:16.134 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46685.log: gzip 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46685.log.gz 2024-09-14T06:10:16.134 INFO:teuthology.orchestra.run.smithi191.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.admin.41784.log 2024-09-14T06:10:16.134 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57985.log 2024-09-14T06:10:16.134 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.41784.log: /var/log/ceph/cluster2-client.mirror.37683.log: gzip 3.5% -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70771.log 2024-09-14T06:10:16.134 INFO:teuthology.orchestra.run.smithi191.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.41784.log.gz 2024-09-14T06:10:16.134 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.37317.log.gz 2024-09-14T06:10:16.134 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43962.log 2024-09-14T06:10:16.134 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.38561.log.gz 2024-09-14T06:10:16.134 INFO:teuthology.orchestra.run.smithi191.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.37683.log.gz 2024-09-14T06:10:16.134 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.57985.log: /var/log/ceph/cluster2-client.mirror.70771.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.57985.log.gz 2024-09-14T06:10:16.134 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52089.log 2024-09-14T06:10:16.134 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.43962.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.70771.log.gz 2024-09-14T06:10:16.134 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42009.log 2024-09-14T06:10:16.135 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.52089.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.52089.log.gz 2024-09-14T06:10:16.135 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57534.log 2024-09-14T06:10:16.135 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.42009.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.42009.log.gz 2024-09-14T06:10:16.135 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70381.log 2024-09-14T06:10:16.135 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.57534.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43962.log.gz 2024-09-14T06:10:16.135 INFO:teuthology.orchestra.run.smithi191.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57534.log.gz 2024-09-14T06:10:16.135 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62028.log 2024-09-14T06:10:16.135 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.70381.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.70381.log.gz 2024-09-14T06:10:16.135 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49270.log 2024-09-14T06:10:16.135 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36069.log 2024-09-14T06:10:16.135 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49270.log: /var/log/ceph/cluster1-client.mirror.62028.log: 76.9% -- replaced with /var/log/ceph/cluster2-client.mirror.49270.log.gz 2024-09-14T06:10:16.135 INFO:teuthology.orchestra.run.smithi191.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.62028.log.gz 2024-09-14T06:10:16.135 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44306.log 2024-09-14T06:10:16.135 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.36069.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36069.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41066.log 2024-09-14T06:10:16.136 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-14T06:10:16.136 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52041.log 2024-09-14T06:10:16.136 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.41066.log: /var/log/ceph/cluster2-client.mirror.44306.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.41066.log.gz 2024-09-14T06:10:16.136 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71263.log 2024-09-14T06:10:16.136 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.52041.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.52041.log.gz 2024-09-14T06:10:16.136 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54973.log 2024-09-14T06:10:16.136 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.71263.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.71263.log.gz 2024-09-14T06:10:16.136 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67458.log 2024-09-14T06:10:16.136 INFO:teuthology.orchestra.run.smithi191.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44306.log.gz 2024-09-14T06:10:16.136 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.54973.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58074.log 2024-09-14T06:10:16.136 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.67458.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67458.log.gz 2024-09-14T06:10:16.136 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42414.log 2024-09-14T06:10:16.136 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.58074.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54973.log.gz 2024-09-14T06:10:16.136 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42105.log 2024-09-14T06:10:16.137 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.42414.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.42414.log.gz 2024-09-14T06:10:16.137 INFO:teuthology.orchestra.run.smithi191.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58074.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67660.log 2024-09-14T06:10:16.137 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-14T06:10:16.137 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44068.log 2024-09-14T06:10:16.144 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.42105.log: /var/log/ceph/cluster1-client.mirror.67660.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.42105.log.gz 2024-09-14T06:10:16.144 INFO:teuthology.orchestra.run.smithi191.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67660.log.gz 2024-09-14T06:10:16.144 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55651.log 2024-09-14T06:10:16.144 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.44068.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54533.log 2024-09-14T06:10:16.144 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.55651.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.55651.log.gz 2024-09-14T06:10:16.144 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.46781.log 2024-09-14T06:10:16.144 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.54533.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.54533.log.gz 2024-09-14T06:10:16.144 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60918.log 2024-09-14T06:10:16.144 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.3.46781.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44068.log.gz 2024-09-14T06:10:16.144 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44134.log 2024-09-14T06:10:16.145 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.60918.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60918.log.gz 2024-09-14T06:10:16.145 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49418.log 2024-09-14T06:10:16.145 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.44134.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44134.log.gz 2024-09-14T06:10:16.145 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66378.log 2024-09-14T06:10:16.145 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49418.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42796.log 2024-09-14T06:10:16.145 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.66378.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66378.log.gz 2024-09-14T06:10:16.152 INFO:teuthology.orchestra.run.smithi191.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48821.log.gz 2024-09-14T06:10:16.152 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57493.log 2024-09-14T06:10:16.155 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.42796.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42796.log.gz 2024-09-14T06:10:16.155 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66693.log 2024-09-14T06:10:16.158 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.57493.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.57493.log.gz 2024-09-14T06:10:16.159 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66558.log 2024-09-14T06:10:16.159 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.66693.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66693.log.gz 2024-09-14T06:10:16.159 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50172.log 2024-09-14T06:10:16.159 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.66558.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66558.log.gz 2024-09-14T06:10:16.159 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52941.log 2024-09-14T06:10:16.159 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.50172.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50172.log.gz 2024-09-14T06:10:16.161 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39415.log 2024-09-14T06:10:16.162 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.52941.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52941.log.gz 2024-09-14T06:10:16.162 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46887.log 2024-09-14T06:10:16.162 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.39415.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36449.log 2024-09-14T06:10:16.166 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46887.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46887.log.gz 2024-09-14T06:10:16.166 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66043.log 2024-09-14T06:10:16.166 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.36449.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36449.log.gz 2024-09-14T06:10:16.166 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66873.log 2024-09-14T06:10:16.166 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.66043.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.66043.log.gz 2024-09-14T06:10:16.166 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65477.log 2024-09-14T06:10:16.166 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.66873.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66873.log.gz 2024-09-14T06:10:16.167 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50919.log 2024-09-14T06:10:16.169 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65477.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65477.log.gz 2024-09-14T06:10:16.169 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55603.log 2024-09-14T06:10:16.170 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.50919.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50919.log.gz 2024-09-14T06:10:16.171 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52198.log 2024-09-14T06:10:16.172 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.55603.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55603.log.gz 2024-09-14T06:10:16.172 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61077.log 2024-09-14T06:10:16.174 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.52198.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52198.log.gz 2024-09-14T06:10:16.174 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44349.log 2024-09-14T06:10:16.175 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.61077.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.61077.log.gz 2024-09-14T06:10:16.175 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46321.log 2024-09-14T06:10:16.182 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.44349.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44349.log.gz 2024-09-14T06:10:16.182 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49663.log 2024-09-14T06:10:16.182 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46321.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46321.log.gz 2024-09-14T06:10:16.183 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41270.log 2024-09-14T06:10:16.204 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49663.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63269.log 2024-09-14T06:10:16.209 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.41270.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41270.log.gz 2024-09-14T06:10:16.210 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39006.log 2024-09-14T06:10:16.210 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.63269.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63269.log.gz 2024-09-14T06:10:16.219 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48999.log 2024-09-14T06:10:16.222 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.39006.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.39006.log.gz 2024-09-14T06:10:16.232 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69155.log 2024-09-14T06:10:16.232 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48999.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48999.log.gz 2024-09-14T06:10:16.243 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57035.log 2024-09-14T06:10:16.243 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.69155.log: 81.5% -- replaced with /var/log/ceph/cluster2-client.mirror.69155.log.gz 2024-09-14T06:10:16.254 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52218.log 2024-09-14T06:10:16.254 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.57035.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57035.log.gz 2024-09-14T06:10:16.266 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51905.log 2024-09-14T06:10:16.266 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.52218.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52218.log.gz 2024-09-14T06:10:16.276 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61014.log 2024-09-14T06:10:16.290 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.51905.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.39435.log.gz 2024-09-14T06:10:16.290 INFO:teuthology.orchestra.run.smithi191.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.51905.log.gz 2024-09-14T06:10:16.291 INFO:teuthology.orchestra.run.smithi191.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.39415.log.gz 2024-09-14T06:10:16.295 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43510.log 2024-09-14T06:10:16.295 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.61014.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.61014.log.gz 2024-09-14T06:10:16.304 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56343.log 2024-09-14T06:10:16.304 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.43510.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43510.log.gz 2024-09-14T06:10:16.336 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.36759.log 2024-09-14T06:10:16.339 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.56343.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56343.log.gz 2024-09-14T06:10:16.339 INFO:teuthology.orchestra.run.smithi191.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46661.log.gz 2024-09-14T06:10:16.339 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64014.log 2024-09-14T06:10:16.339 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.1.36759.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61522.log 2024-09-14T06:10:16.341 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64014.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64014.log.gz 2024-09-14T06:10:16.341 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.71657.log 2024-09-14T06:10:16.341 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.61522.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.61522.log.gz 2024-09-14T06:10:16.342 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56736.log 2024-09-14T06:10:16.343 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.2.71657.log: 91.6% -- replaced with /var/log/ceph/cluster1-client.mirror.2.71657.log.gz 2024-09-14T06:10:16.345 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59635.log 2024-09-14T06:10:16.345 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.56736.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.56736.log.gz 2024-09-14T06:10:16.347 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51864.log 2024-09-14T06:10:16.347 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.59635.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.59635.log.gz 2024-09-14T06:10:16.347 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66243.log 2024-09-14T06:10:16.350 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.51864.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.51864.log.gz 2024-09-14T06:10:16.350 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72379.log 2024-09-14T06:10:16.350 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.66243.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66243.log.gz 2024-09-14T06:10:16.350 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67818.log 2024-09-14T06:10:16.350 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.72379.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72379.log.gz 2024-09-14T06:10:16.350 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54815.log 2024-09-14T06:10:16.353 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.67818.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67818.log.gz 2024-09-14T06:10:16.353 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62302.log 2024-09-14T06:10:16.353 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.54815.log: 82.1% -- replaced with /var/log/ceph/cluster1-client.mirror.54815.log.gz 2024-09-14T06:10:16.353 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66580.log 2024-09-14T06:10:16.355 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.62302.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62302.log.gz 2024-09-14T06:10:16.356 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56540.log 2024-09-14T06:10:16.356 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.66580.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66580.log.gz 2024-09-14T06:10:16.357 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47920.log 2024-09-14T06:10:16.359 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.56540.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56540.log.gz 2024-09-14T06:10:16.359 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57917.log 2024-09-14T06:10:16.370 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47920.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53548.log 2024-09-14T06:10:16.371 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.57917.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.57917.log.gz 2024-09-14T06:10:16.374 INFO:teuthology.orchestra.run.smithi191.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.1.36759.log.gz 2024-09-14T06:10:16.381 INFO:teuthology.orchestra.run.smithi191.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48476.log.gz 2024-09-14T06:10:16.381 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52740.log 2024-09-14T06:10:16.382 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.53548.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53548.log.gz 2024-09-14T06:10:16.384 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58773.log 2024-09-14T06:10:16.386 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.52740.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52740.log.gz 2024-09-14T06:10:16.386 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36849.log 2024-09-14T06:10:16.390 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.58773.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.58773.log.gz 2024-09-14T06:10:16.390 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54605.log 2024-09-14T06:10:16.390 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.36849.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50851.log 2024-09-14T06:10:16.390 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.54605.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.54605.log.gz 2024-09-14T06:10:16.390 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62726.log 2024-09-14T06:10:16.395 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.50851.log: 74.2% -- replaced with /var/log/ceph/cluster2-client.mirror.36849.log.gz 2024-09-14T06:10:16.397 INFO:teuthology.orchestra.run.smithi191.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.50851.log.gz 2024-09-14T06:10:16.397 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48979.log 2024-09-14T06:10:16.397 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.62726.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62726.log.gz 2024-09-14T06:10:16.398 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51973.log 2024-09-14T06:10:16.412 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48979.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50602.log 2024-09-14T06:10:16.414 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.51973.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51973.log.gz 2024-09-14T06:10:16.416 INFO:teuthology.orchestra.run.smithi191.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47760.log.gz 2024-09-14T06:10:16.419 INFO:teuthology.orchestra.run.smithi143.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-mon.a.log.gz 2024-09-14T06:10:16.423 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50489.log 2024-09-14T06:10:16.425 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.50602.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.50602.log.gz 2024-09-14T06:10:16.436 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42057.log 2024-09-14T06:10:16.436 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.50489.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.50489.log.gz 2024-09-14T06:10:16.439 INFO:teuthology.orchestra.run.smithi191.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49663.log.gz 2024-09-14T06:10:16.439 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62510.log 2024-09-14T06:10:16.439 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.42057.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.45448.log 2024-09-14T06:10:16.440 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.62510.log: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.42057.log.gz 2024-09-14T06:10:16.440 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64733.log 2024-09-14T06:10:16.441 INFO:teuthology.orchestra.run.smithi191.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.62510.log.gz 2024-09-14T06:10:16.441 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.1.45448.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66153.log 2024-09-14T06:10:16.444 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64733.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64733.log.gz 2024-09-14T06:10:16.444 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57317.log 2024-09-14T06:10:16.444 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.66153.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.66153.log.gz 2024-09-14T06:10:16.445 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52805.log 2024-09-14T06:10:16.447 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.57317.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57317.log.gz 2024-09-14T06:10:16.447 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41290.log 2024-09-14T06:10:16.450 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.52805.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52805.log.gz 2024-09-14T06:10:16.450 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.71300.log 2024-09-14T06:10:16.456 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.41290.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.41290.log.gz 2024-09-14T06:10:16.457 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36323.log 2024-09-14T06:10:16.460 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.1.71300.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.1.71300.log.gz 2024-09-14T06:10:16.460 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68088.log 2024-09-14T06:10:16.460 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.36323.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36323.log.gz 2024-09-14T06:10:16.462 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69074.log 2024-09-14T06:10:16.462 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.68088.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68088.log.gz 2024-09-14T06:10:16.462 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63559.log 2024-09-14T06:10:16.463 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.69074.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.69074.log.gz 2024-09-14T06:10:16.463 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54750.log 2024-09-14T06:10:16.465 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.63559.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63559.log.gz 2024-09-14T06:10:16.465 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62664.log 2024-09-14T06:10:16.466 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.54750.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54750.log.gz 2024-09-14T06:10:16.466 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45554.log 2024-09-14T06:10:16.468 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.62664.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62664.log.gz 2024-09-14T06:10:16.468 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48779.log 2024-09-14T06:10:16.470 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45554.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45554.log.gz 2024-09-14T06:10:16.470 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62486.log 2024-09-14T06:10:16.471 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48779.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48779.log.gz 2024-09-14T06:10:16.472 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61751.log 2024-09-14T06:10:16.473 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.62486.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.62486.log.gz 2024-09-14T06:10:16.473 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67053.log 2024-09-14T06:10:16.475 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.61751.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.61751.log.gz 2024-09-14T06:10:16.475 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67750.log 2024-09-14T06:10:16.475 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.67053.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67053.log.gz 2024-09-14T06:10:16.476 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59119.log 2024-09-14T06:10:16.478 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.67750.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67750.log.gz 2024-09-14T06:10:16.478 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70991.log 2024-09-14T06:10:16.478 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.59119.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59119.log.gz 2024-09-14T06:10:16.478 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59432.log 2024-09-14T06:10:16.479 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.70991.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.70991.log.gz 2024-09-14T06:10:16.479 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62555.log 2024-09-14T06:10:16.481 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.59432.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59432.log.gz 2024-09-14T06:10:16.481 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63331.log 2024-09-14T06:10:16.482 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.62555.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.62555.log.gz 2024-09-14T06:10:16.483 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42440.log 2024-09-14T06:10:16.484 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.63331.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63331.log.gz 2024-09-14T06:10:16.484 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41334.log 2024-09-14T06:10:16.487 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.42440.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.42440.log.gz 2024-09-14T06:10:16.487 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54726.log 2024-09-14T06:10:16.505 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.41334.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46443.log 2024-09-14T06:10:16.506 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.54726.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54726.log.gz 2024-09-14T06:10:16.524 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71475.log 2024-09-14T06:10:16.525 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46443.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46443.log.gz 2024-09-14T06:10:16.533 INFO:teuthology.orchestra.run.smithi191.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.39479.log.gz 2024-09-14T06:10:16.535 INFO:teuthology.orchestra.run.smithi191.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.3.46781.log.gz 2024-09-14T06:10:16.536 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58096.log 2024-09-14T06:10:16.539 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.71475.log: 91.1% -- replaced with /var/log/ceph/cluster2-client.mirror.71475.log.gz 2024-09-14T06:10:16.539 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61796.log 2024-09-14T06:10:16.540 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.58096.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.58096.log.gz 2024-09-14T06:10:16.541 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57703.log 2024-09-14T06:10:16.543 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.61796.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61796.log.gz 2024-09-14T06:10:16.543 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61655.log 2024-09-14T06:10:16.545 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.57703.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.57703.log.gz 2024-09-14T06:10:16.545 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67345.log 2024-09-14T06:10:16.546 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.61655.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61655.log.gz 2024-09-14T06:10:16.547 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.42619.log 2024-09-14T06:10:16.547 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.67345.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67345.log.gz 2024-09-14T06:10:16.548 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51768.log 2024-09-14T06:10:16.569 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.3.42619.log: 94.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53343.log 2024-09-14T06:10:16.569 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.51768.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51768.log.gz 2024-09-14T06:10:16.569 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.3.42619.log.gz 2024-09-14T06:10:16.569 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62891.log 2024-09-14T06:10:16.570 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.53343.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53343.log.gz 2024-09-14T06:10:16.570 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50963.log 2024-09-14T06:10:16.574 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.62891.log: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.62891.log.gz 2024-09-14T06:10:16.574 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64754.log 2024-09-14T06:10:16.575 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.50963.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.50963.log.gz 2024-09-14T06:10:16.575 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.63052.log 2024-09-14T06:10:16.576 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64754.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64754.log.gz 2024-09-14T06:10:16.576 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48136.log 2024-09-14T06:10:16.580 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.0.63052.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.0.63052.log.gz 2024-09-14T06:10:16.580 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55338.log 2024-09-14T06:10:16.581 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48136.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48136.log.gz 2024-09-14T06:10:16.581 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63476.log 2024-09-14T06:10:16.582 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.55338.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55338.log.gz 2024-09-14T06:10:16.582 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68245.log 2024-09-14T06:10:16.585 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.63476.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63476.log.gz 2024-09-14T06:10:16.586 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53568.log 2024-09-14T06:10:16.587 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.68245.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68245.log.gz 2024-09-14T06:10:16.587 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52893.log 2024-09-14T06:10:16.589 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.53568.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53568.log.gz 2024-09-14T06:10:16.589 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.2.log 2024-09-14T06:10:16.589 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.52893.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52893.log.gz 2024-09-14T06:10:16.590 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47293.log 2024-09-14T06:10:16.591 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57614.log 2024-09-14T06:10:16.591 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.47293.log: 86.4% -- replaced with /var/log/ceph/cluster1-client.mirror.47293.log.gz 2024-09-14T06:10:16.599 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66603.log 2024-09-14T06:10:16.600 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49016.log 2024-09-14T06:10:16.600 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.57614.log: /var/log/ceph/cluster1-client.mirror.66603.log: 87.4% 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.66603.log.gz 2024-09-14T06:10:16.602 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51294.log 2024-09-14T06:10:16.602 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.57614.log.gz 2024-09-14T06:10:16.602 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49016.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49016.log.gz 2024-09-14T06:10:16.603 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64442.log 2024-09-14T06:10:16.603 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.51294.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51294.log.gz 2024-09-14T06:10:16.616 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60207.log 2024-09-14T06:10:16.617 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64442.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64442.log.gz 2024-09-14T06:10:16.620 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53751.log 2024-09-14T06:10:16.623 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.60207.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60207.log.gz 2024-09-14T06:10:16.626 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71455.log 2024-09-14T06:10:16.632 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.53751.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53751.log.gz 2024-09-14T06:10:16.633 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65333.log 2024-09-14T06:10:16.636 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.71455.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.71455.log.gz 2024-09-14T06:10:16.638 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55073.log 2024-09-14T06:10:16.638 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65333.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65333.log.gz 2024-09-14T06:10:16.638 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55475.log 2024-09-14T06:10:16.642 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.55073.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55073.log.gz 2024-09-14T06:10:16.642 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65935.log 2024-09-14T06:10:16.644 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.55475.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55475.log.gz 2024-09-14T06:10:16.671 INFO:teuthology.orchestra.run.smithi191.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48158.log.gz 2024-09-14T06:10:16.671 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63807.log 2024-09-14T06:10:16.672 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65935.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65935.log.gz 2024-09-14T06:10:16.673 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64303.log 2024-09-14T06:10:16.674 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.63807.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63807.log.gz 2024-09-14T06:10:16.674 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35942.log 2024-09-14T06:10:16.681 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64303.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64303.log.gz 2024-09-14T06:10:16.681 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46460.log 2024-09-14T06:10:16.681 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56874.log 2024-09-14T06:10:16.681 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.35942.log: /var/log/ceph/cluster2-client.mirror.46460.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35942.log.gz 2024-09-14T06:10:16.681 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61219.log 2024-09-14T06:10:16.681 INFO:teuthology.orchestra.run.smithi191.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46460.log.gz 2024-09-14T06:10:16.681 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.56874.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67390.log 2024-09-14T06:10:16.681 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.61219.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.56874.log.gz 2024-09-14T06:10:16.682 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38794.log 2024-09-14T06:10:16.682 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.67390.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67390.log.gz 2024-09-14T06:10:16.682 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45837.log 2024-09-14T06:10:16.682 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.38794.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44858.log 2024-09-14T06:10:16.682 INFO:teuthology.orchestra.run.smithi191.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.38794.log.gz 2024-09-14T06:10:16.690 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45837.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68869.log 2024-09-14T06:10:16.694 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.44858.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44858.log.gz 2024-09-14T06:10:16.707 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68719.log 2024-09-14T06:10:16.709 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.68869.log: 94.2% -- replaced with /var/log/ceph/cluster2-client.mirror.1.45448.log.gz 2024-09-14T06:10:16.710 INFO:teuthology.orchestra.run.smithi191.stderr: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.68869.log.gz 2024-09-14T06:10:16.710 INFO:teuthology.orchestra.run.smithi191.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.61219.log.gz 2024-09-14T06:10:16.719 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62209.log 2024-09-14T06:10:16.719 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.68719.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.68719.log.gz 2024-09-14T06:10:16.721 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44919.log 2024-09-14T06:10:16.721 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.62209.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62209.log.gz 2024-09-14T06:10:16.721 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46582.log 2024-09-14T06:10:16.721 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68110.log 2024-09-14T06:10:16.726 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46582.log: /var/log/ceph/cluster2-client.mirror.44919.log: 90.9% -- replaced with /var/log/ceph/cluster2-client.mirror.44919.log.gz 2024-09-14T06:10:16.726 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71219.log 2024-09-14T06:10:16.726 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.68110.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.68110.log.gz 2024-09-14T06:10:16.727 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59748.log 2024-09-14T06:10:16.727 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.71219.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.71219.log.gz 2024-09-14T06:10:16.728 INFO:teuthology.orchestra.run.smithi191.stderr:gzip/var/log/ceph/cluster1-client.mirror.59748.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59748.log.gz 2024-09-14T06:10:16.731 INFO:teuthology.orchestra.run.smithi191.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61980.log 2024-09-14T06:10:16.741 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70185.log 2024-09-14T06:10:16.745 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.61980.log: 87.4% 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47920.log.gz 2024-09-14T06:10:16.748 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44878.log 2024-09-14T06:10:16.749 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.70185.log: 82.9% -- replaced with /var/log/ceph/cluster2-client.mirror.70185.log.gz 2024-09-14T06:10:16.750 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69494.log 2024-09-14T06:10:16.750 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.44878.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67188.log 2024-09-14T06:10:16.752 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.69494.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44878.log.gz 2024-09-14T06:10:16.752 INFO:teuthology.orchestra.run.smithi191.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.69494.log.gz 2024-09-14T06:10:16.752 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.61980.log.gz 2024-09-14T06:10:16.752 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49877.log 2024-09-14T06:10:16.752 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.67188.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53166.log 2024-09-14T06:10:16.754 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49877.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49877.log.gz 2024-09-14T06:10:16.754 INFO:teuthology.orchestra.run.smithi191.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67188.log.gz 2024-09-14T06:10:16.754 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69457.log 2024-09-14T06:10:16.756 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.53166.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.53166.log.gz 2024-09-14T06:10:16.756 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43919.log 2024-09-14T06:10:16.756 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.69457.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52467.log 2024-09-14T06:10:16.756 INFO:teuthology.orchestra.run.smithi191.stderr: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.69457.log.gz 2024-09-14T06:10:16.756 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.43919.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53640.log 2024-09-14T06:10:16.756 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.52467.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.52467.log.gz 2024-09-14T06:10:16.762 INFO:teuthology.orchestra.run.smithi191.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49418.log.gz 2024-09-14T06:10:16.764 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44964.log 2024-09-14T06:10:16.764 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.53640.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53640.log.gz 2024-09-14T06:10:16.769 INFO:teuthology.orchestra.run.smithi191.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43919.log.gz 2024-09-14T06:10:16.769 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50288.log 2024-09-14T06:10:16.769 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.44964.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.44964.log.gz 2024-09-14T06:10:16.770 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53052.log 2024-09-14T06:10:16.774 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose --/var/log/ceph/cluster1-client.mirror.50288.log: /var/log/ceph/cluster2-client.mirror.59037.log 2024-09-14T06:10:16.774 INFO:teuthology.orchestra.run.smithi191.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.50288.log.gz 2024-09-14T06:10:16.774 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.53052.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69374.log 2024-09-14T06:10:16.774 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.59037.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59861.log 2024-09-14T06:10:16.774 INFO:teuthology.orchestra.run.smithi191.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53052.log.gz 2024-09-14T06:10:16.774 INFO:teuthology.orchestra.run.smithi191.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.59037.log.gz 2024-09-14T06:10:16.774 INFO:teuthology.orchestra.run.smithi191.stderr:gzip/var/log/ceph/cluster1-client.mirror.69374.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.36886.log 2024-09-14T06:10:16.774 INFO:teuthology.orchestra.run.smithi191.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69374.log.gz 2024-09-14T06:10:16.786 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.59861.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59861.log.gz 2024-09-14T06:10:16.786 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.41874.log 2024-09-14T06:10:16.786 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.1.36886.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54364.log 2024-09-14T06:10:16.786 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.3.41874.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71533.log 2024-09-14T06:10:16.786 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.54364.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53954.log 2024-09-14T06:10:16.786 INFO:teuthology.orchestra.run.smithi191.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.54364.log.gz 2024-09-14T06:10:16.786 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.71533.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41360.log 2024-09-14T06:10:16.786 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.53954.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.3.41874.log.gz 2024-09-14T06:10:16.786 INFO:teuthology.orchestra.run.smithi191.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.71533.log.gz 2024-09-14T06:10:16.786 INFO:teuthology.orchestra.run.smithi191.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.53954.log.gz 2024-09-14T06:10:16.787 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45795.log 2024-09-14T06:10:16.787 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56186.log 2024-09-14T06:10:16.787 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.41360.log: /var/log/ceph/cluster2-client.mirror.45795.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45633.log 2024-09-14T06:10:16.787 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.56186.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.41360.log.gz 2024-09-14T06:10:16.787 INFO:teuthology.orchestra.run.smithi191.stderr: 89.5%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61956.log 2024-09-14T06:10:16.787 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.45795.log.gz 2024-09-14T06:10:16.787 INFO:teuthology.orchestra.run.smithi191.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.56186.log.gz 2024-09-14T06:10:16.787 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32540.log 2024-09-14T06:10:16.787 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45633.log: /var/log/ceph/cluster1-client.mirror.61956.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61956.log.gz 2024-09-14T06:10:16.787 INFO:teuthology.orchestra.run.smithi191.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45633.log.gz 2024-09-14T06:10:16.787 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59384.log 2024-09-14T06:10:16.788 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.32540.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51660.log 2024-09-14T06:10:16.788 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.32540.log.gz 2024-09-14T06:10:16.788 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48620.log 2024-09-14T06:10:16.788 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.59384.log: /var/log/ceph/cluster2-client.mirror.51660.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.59384.log.gz 2024-09-14T06:10:16.788 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52335.log 2024-09-14T06:10:16.788 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48662.log 2024-09-14T06:10:16.788 INFO:teuthology.orchestra.run.smithi191.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51660.log.gz 2024-09-14T06:10:16.788 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.52335.log: /var/log/ceph/cluster2-client.mirror.48620.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48620.log.gz 2024-09-14T06:10:16.788 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37266.log 2024-09-14T06:10:16.788 INFO:teuthology.orchestra.run.smithi191.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52335.log.gz 2024-09-14T06:10:16.788 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48662.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53098.log 2024-09-14T06:10:16.789 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.37266.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61243.log 2024-09-14T06:10:16.789 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.53098.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69097.log 2024-09-14T06:10:16.792 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.61243.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53098.log.gz 2024-09-14T06:10:16.792 INFO:teuthology.orchestra.run.smithi191.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61243.log.gz 2024-09-14T06:10:16.792 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47214.log 2024-09-14T06:10:16.792 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.69097.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.69097.log.gz 2024-09-14T06:10:16.792 INFO:teuthology.orchestra.run.smithi191.stderr: 79.9% -- replaced with /var/log/ceph/cluster1-client.admin.37266.log.gz 2024-09-14T06:10:16.792 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47840.log 2024-09-14T06:10:16.792 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47214.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47214.log.gz 2024-09-14T06:10:16.792 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55362.log 2024-09-14T06:10:16.797 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47840.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47520.log.gz 2024-09-14T06:10:16.798 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47135.log 2024-09-14T06:10:16.798 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.55362.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55362.log.gz 2024-09-14T06:10:16.802 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59336.log 2024-09-14T06:10:16.803 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47135.log: 92.8% -- replaced with /var/log/ceph/cluster2-client.mirror.1.36886.log.gz 2024-09-14T06:10:16.805 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43833.log 2024-09-14T06:10:16.805 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.59336.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59336.log.gz 2024-09-14T06:10:16.805 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43552.log 2024-09-14T06:10:16.807 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.43833.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47135.log.gz 2024-09-14T06:10:16.807 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.36747.log 2024-09-14T06:10:16.812 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.43552.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43833.log.gz 2024-09-14T06:10:16.812 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36533.log 2024-09-14T06:10:16.812 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.0.36747.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59167.log 2024-09-14T06:10:16.813 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.36533.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36533.log.gz 2024-09-14T06:10:16.814 INFO:teuthology.orchestra.run.smithi191.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43552.log.gz 2024-09-14T06:10:16.818 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67728.log 2024-09-14T06:10:16.818 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.59167.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48037.log 2024-09-14T06:10:16.818 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.67728.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59167.log.gz 2024-09-14T06:10:16.818 INFO:teuthology.orchestra.run.smithi191.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67728.log.gz 2024-09-14T06:10:16.818 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61055.log 2024-09-14T06:10:16.818 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48037.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47067.log 2024-09-14T06:10:16.819 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.61055.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48037.log.gz 2024-09-14T06:10:16.820 INFO:teuthology.orchestra.run.smithi191.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.61055.log.gz 2024-09-14T06:10:16.821 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57252.log 2024-09-14T06:10:16.823 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47067.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40888.log 2024-09-14T06:10:16.823 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.57252.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47067.log.gz 2024-09-14T06:10:16.823 INFO:teuthology.orchestra.run.smithi191.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57252.log.gz 2024-09-14T06:10:16.824 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69582.log 2024-09-14T06:10:16.824 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.40888.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54952.log 2024-09-14T06:10:16.824 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.69582.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.40888.log.gz 2024-09-14T06:10:16.826 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49991.log 2024-09-14T06:10:16.826 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.54952.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.54952.log.gz 2024-09-14T06:10:16.826 INFO:teuthology.orchestra.run.smithi191.stderr: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69582.log.gz 2024-09-14T06:10:16.826 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39533.log 2024-09-14T06:10:16.826 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.49991.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.49991.log.gz 2024-09-14T06:10:16.829 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72307.log 2024-09-14T06:10:16.830 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.39533.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72803.log 2024-09-14T06:10:16.831 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.72307.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.72307.log.gz 2024-09-14T06:10:16.833 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.39533.log.gz 2024-09-14T06:10:16.840 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64775.log 2024-09-14T06:10:16.840 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.72803.log: gzip -5 --verbose -- 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72803.log.gz 2024-09-14T06:10:16.840 INFO:teuthology.orchestra.run.smithi191.stderr: /var/log/ceph/cluster2-client.admin.36491.log 2024-09-14T06:10:16.840 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64775.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37522.log 2024-09-14T06:10:16.840 INFO:teuthology.orchestra.run.smithi191.stderr: 88.4%/var/log/ceph/cluster2-client.admin.36491.log: -- replaced with /var/log/ceph/cluster2-client.mirror.64775.log.gz 2024-09-14T06:10:16.840 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36491.log.gz 2024-09-14T06:10:16.840 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46205.log 2024-09-14T06:10:16.840 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.37522.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.0.36747.log.gz 2024-09-14T06:10:16.841 INFO:teuthology.orchestra.run.smithi191.stderr: 54.4% -- replaced with /var/log/ceph/cluster2-client.mirror.37522.log.gz 2024-09-14T06:10:16.841 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51390.log 2024-09-14T06:10:16.841 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46205.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36155.log 2024-09-14T06:10:16.841 INFO:teuthology.orchestra.run.smithi191.stderr: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46205.log.gz 2024-09-14T06:10:16.841 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.51390.log: gzip -5 --verbose -- 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.51390.log.gz 2024-09-14T06:10:16.841 INFO:teuthology.orchestra.run.smithi191.stderr: /var/log/ceph/cluster2-client.mirror.65997.log 2024-09-14T06:10:16.841 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.36155.log: 0.0%gzip -- replaced with /var/log/ceph/cluster2-client.admin.36155.log.gz -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55294.log 2024-09-14T06:10:16.841 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-14T06:10:16.841 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65997.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37385.log 2024-09-14T06:10:16.841 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.55294.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65997.log.gz 2024-09-14T06:10:16.841 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57816.log 2024-09-14T06:10:16.841 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.37385.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.55294.log.gz 2024-09-14T06:10:16.841 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69118.log 2024-09-14T06:10:16.842 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.57816.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60529.log 2024-09-14T06:10:16.843 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.69118.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69118.log.gz 2024-09-14T06:10:16.843 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46364.log 2024-09-14T06:10:16.843 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.60529.log: 54.5% -- replaced with /var/log/ceph/cluster1-client.mirror.37385.log.gz 2024-09-14T06:10:16.844 INFO:teuthology.orchestra.run.smithi191.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.57816.log.gz 2024-09-14T06:10:16.844 INFO:teuthology.orchestra.run.smithi191.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60529.log.gz 2024-09-14T06:10:16.845 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55844.log 2024-09-14T06:10:16.845 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46364.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46907.log 2024-09-14T06:10:16.845 INFO:teuthology.orchestra.run.smithi191.stderr: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46364.log.gz 2024-09-14T06:10:16.847 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.55844.log: gzip -5 --verbose 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55844.log.gz 2024-09-14T06:10:16.847 INFO:teuthology.orchestra.run.smithi191.stderr: -- /var/log/ceph/cluster1-client.mirror.61268.log 2024-09-14T06:10:16.848 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46907.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65457.log 2024-09-14T06:10:16.848 INFO:teuthology.orchestra.run.smithi191.stderr: 87.5%/var/log/ceph/cluster1-client.mirror.61268.log: -- replaced with /var/log/ceph/cluster2-client.mirror.46907.log.gz 2024-09-14T06:10:16.848 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39933.log 2024-09-14T06:10:16.848 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65457.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65457.log.gz 2024-09-14T06:10:16.848 INFO:teuthology.orchestra.run.smithi191.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.61268.log.gz 2024-09-14T06:10:16.849 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71513.log 2024-09-14T06:10:16.849 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.39933.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64463.log 2024-09-14T06:10:16.850 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.71513.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.71513.log.gz 2024-09-14T06:10:16.852 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66333.log 2024-09-14T06:10:16.855 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64463.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54904.log 2024-09-14T06:10:16.855 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.66333.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64463.log.gz 2024-09-14T06:10:16.855 INFO:teuthology.orchestra.run.smithi191.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66333.log.gz 2024-09-14T06:10:16.855 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54244.log 2024-09-14T06:10:16.855 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.54904.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42033.log 2024-09-14T06:10:16.855 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.54244.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54904.log.gz 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54244.log.gz 2024-09-14T06:10:16.855 INFO:teuthology.orchestra.run.smithi191.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.39933.log.gz 2024-09-14T06:10:16.856 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-14T06:10:16.856 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67840.log 2024-09-14T06:10:16.863 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.42033.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56447.log 2024-09-14T06:10:16.864 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.67840.log: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.42033.log.gz 2024-09-14T06:10:16.864 INFO:teuthology.orchestra.run.smithi191.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.67840.log.gz 2024-09-14T06:10:16.864 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39304.log 2024-09-14T06:10:16.864 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.56447.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55969.log 2024-09-14T06:10:16.864 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.39304.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56447.log.gz 2024-09-14T06:10:16.864 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35558.log 2024-09-14T06:10:16.864 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.55969.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47738.log 2024-09-14T06:10:16.864 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.35558.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66963.log 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.55969.log.gz 2024-09-14T06:10:16.864 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-14T06:10:16.864 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47738.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44838.log 2024-09-14T06:10:16.864 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47738.log.gz/var/log/ceph/cluster1-client.mirror.66963.log: 2024-09-14T06:10:16.864 INFO:teuthology.orchestra.run.smithi191.stderr: 87.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69668.log 2024-09-14T06:10:16.865 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.44838.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44838.log.gz 2024-09-14T06:10:16.865 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.66963.log.gz 2024-09-14T06:10:16.865 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48957.log 2024-09-14T06:10:16.865 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.69668.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69668.log.gz 2024-09-14T06:10:16.865 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39221.log 2024-09-14T06:10:16.865 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48957.log: 82.1% -- replaced with /var/log/ceph/cluster2-client.mirror.39304.log.gz 2024-09-14T06:10:16.865 INFO:teuthology.orchestra.run.smithi191.stderr: 80.3% -- replaced with /var/log/ceph/cluster2-client.admin.35558.log.gz 2024-09-14T06:10:16.865 INFO:teuthology.orchestra.run.smithi191.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48957.log.gz 2024-09-14T06:10:16.865 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45026.log 2024-09-14T06:10:16.865 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.39221.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53841.log 2024-09-14T06:10:16.869 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36407.log 2024-09-14T06:10:16.870 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45026.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45026.log.gz 2024-09-14T06:10:16.870 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55121.log 2024-09-14T06:10:16.870 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.36407.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36407.log.gz 2024-09-14T06:10:16.870 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59095.log 2024-09-14T06:10:16.870 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.55121.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55121.log.gz 2024-09-14T06:10:16.870 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51186.log 2024-09-14T06:10:16.870 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.59095.log: /var/log/ceph/cluster1-client.mirror.53841.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59095.log.gz 2024-09-14T06:10:16.870 INFO:teuthology.orchestra.run.smithi191.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53841.log.gz 2024-09-14T06:10:16.870 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45699.log 2024-09-14T06:10:16.873 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.51186.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51186.log.gz 2024-09-14T06:10:16.873 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58853.log 2024-09-14T06:10:16.873 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45699.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.45699.log.gz 2024-09-14T06:10:16.873 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64713.log 2024-09-14T06:10:16.875 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.58853.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40669.log 2024-09-14T06:10:16.875 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64713.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58853.log.gz 2024-09-14T06:10:16.875 INFO:teuthology.orchestra.run.smithi191.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64713.log.gz 2024-09-14T06:10:16.875 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49476.log 2024-09-14T06:10:16.879 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.40669.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44429.log 2024-09-14T06:10:16.879 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49476.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49476.log.gz 2024-09-14T06:10:16.879 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61932.log 2024-09-14T06:10:16.881 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.44429.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.40669.log.gz 2024-09-14T06:10:16.881 INFO:teuthology.orchestra.run.smithi191.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48979.log.gz 2024-09-14T06:10:16.881 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65415.log 2024-09-14T06:10:16.882 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.61932.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.61932.log.gz 2024-09-14T06:10:16.882 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58684.log 2024-09-14T06:10:16.883 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70319.log 2024-09-14T06:10:16.884 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65415.log: /var/log/ceph/cluster1-client.mirror.58684.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58684.log.gz 2024-09-14T06:10:16.884 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61309.log 2024-09-14T06:10:16.884 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.70319.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.70319.log.gz 2024-09-14T06:10:16.885 INFO:teuthology.orchestra.run.smithi191.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65415.log.gz 2024-09-14T06:10:16.886 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.51076.log 2024-09-14T06:10:16.886 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.61309.log: 95.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44429.log.gz 2024-09-14T06:10:16.887 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66715.log 2024-09-14T06:10:16.889 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.1.51076.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61309.log.gz 2024-09-14T06:10:16.889 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45596.log 2024-09-14T06:10:16.889 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.66715.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66715.log.gz 2024-09-14T06:10:16.889 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46988.log 2024-09-14T06:10:16.891 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45596.log: 92.6% -- replaced with /var/log/ceph/cluster1-client.mirror.1.51076.log.gz 2024-09-14T06:10:16.891 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50063.log 2024-09-14T06:10:16.892 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46988.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46988.log.gz 2024-09-14T06:10:16.893 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67480.log 2024-09-14T06:10:16.893 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.50063.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.50063.log.gz 2024-09-14T06:10:16.906 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55434.log 2024-09-14T06:10:16.907 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.67480.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67480.log.gz 2024-09-14T06:10:16.907 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60318.log 2024-09-14T06:10:16.909 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.55434.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.55434.log.gz 2024-09-14T06:10:16.909 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53664.log 2024-09-14T06:10:16.910 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.60318.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.60318.log.gz 2024-09-14T06:10:16.911 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45233.log 2024-09-14T06:10:16.912 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.53664.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53664.log.gz 2024-09-14T06:10:16.912 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53367.log 2024-09-14T06:10:16.912 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45233.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.45233.log.gz 2024-09-14T06:10:16.913 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70671.log 2024-09-14T06:10:16.914 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.53367.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.53367.log.gz 2024-09-14T06:10:16.916 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64940.log 2024-09-14T06:10:16.916 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67210.log 2024-09-14T06:10:16.916 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.70671.log: /var/log/ceph/cluster2-client.mirror.64940.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70671.log.gz 2024-09-14T06:10:16.916 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38654.log 2024-09-14T06:10:16.917 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.67210.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67210.log.gz 2024-09-14T06:10:16.920 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39457.log 2024-09-14T06:10:16.920 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.38654.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70648.log 2024-09-14T06:10:16.921 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.39457.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71998.log 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64940.log.gz 2024-09-14T06:10:16.921 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-14T06:10:16.921 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.70648.log: -- /var/log/ceph/cluster2-client.mirror.51477.log 2024-09-14T06:10:16.921 INFO:teuthology.orchestra.run.smithi191.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70648.log.gz 2024-09-14T06:10:16.921 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49096.log 2024-09-14T06:10:16.922 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.71998.log: /var/log/ceph/cluster2-client.mirror.51477.log: 86.8% 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.71998.log.gz 2024-09-14T06:10:16.922 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.51477.log.gz 2024-09-14T06:10:16.922 INFO:teuthology.orchestra.run.smithi191.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.38654.log.gz 2024-09-14T06:10:16.924 INFO:teuthology.orchestra.run.smithi191.stderr: 89.6%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54144.log 2024-09-14T06:10:16.924 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49096.log: -- replaced with /var/log/ceph/cluster2-client.mirror.39457.log.gz 2024-09-14T06:10:16.924 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69845.log 2024-09-14T06:10:16.924 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.54144.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49096.log.gz 2024-09-14T06:10:16.925 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56061.log 2024-09-14T06:10:16.926 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.69845.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54144.log.gz 2024-09-14T06:10:16.927 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56837.log 2024-09-14T06:10:16.928 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.56061.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56061.log.gz 2024-09-14T06:10:16.928 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52377.log 2024-09-14T06:10:16.928 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.56837.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.69845.log.gz 2024-09-14T06:10:16.930 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57857.log 2024-09-14T06:10:16.930 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.52377.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.56837.log.gz 2024-09-14T06:10:16.930 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47579.log 2024-09-14T06:10:16.932 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.57857.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57857.log.gz 2024-09-14T06:10:16.934 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58295.log 2024-09-14T06:10:16.934 INFO:teuthology.orchestra.run.smithi191.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52377.log.gz 2024-09-14T06:10:16.934 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47579.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42773.log 2024-09-14T06:10:16.934 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.58295.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47579.log.gz 2024-09-14T06:10:16.934 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48454.log 2024-09-14T06:10:16.935 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.42773.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58295.log.gz 2024-09-14T06:10:16.939 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.audit.log 2024-09-14T06:10:16.941 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48454.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42773.log.gz 2024-09-14T06:10:16.941 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65914.log 2024-09-14T06:10:16.942 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2.audit.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57377.log 2024-09-14T06:10:16.942 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65914.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48454.log.gz 2024-09-14T06:10:16.944 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67705.log 2024-09-14T06:10:16.944 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.57377.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65914.log.gz 2024-09-14T06:10:16.944 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59529.log 2024-09-14T06:10:16.944 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.67705.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.57377.log.gz 2024-09-14T06:10:16.944 INFO:teuthology.orchestra.run.smithi191.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.67705.log.gz 2024-09-14T06:10:16.944 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46870.log 2024-09-14T06:10:16.945 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61727.log 2024-09-14T06:10:16.945 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.59529.log: /var/log/ceph/cluster2-client.mirror.46870.log: 82.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59529.log.gz 2024-09-14T06:10:16.946 INFO:teuthology.orchestra.run.smithi191.stderr: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46870.log.gz 2024-09-14T06:10:16.946 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66670.log 2024-09-14T06:10:16.946 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63142.log 2024-09-14T06:10:16.946 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.61727.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.61727.log.gz 2024-09-14T06:10:16.946 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47478.log 2024-09-14T06:10:16.947 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.63142.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.63142.log.gz 2024-09-14T06:10:16.948 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41731.log 2024-09-14T06:10:16.950 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47478.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47478.log.gz 2024-09-14T06:10:16.950 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54405.log 2024-09-14T06:10:16.954 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.41731.log: /var/log/ceph/cluster1-client.mirror.66670.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66670.log.gz 2024-09-14T06:10:16.955 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65064.log 2024-09-14T06:10:16.957 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.54405.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.54405.log.gz 2024-09-14T06:10:16.958 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72331.log 2024-09-14T06:10:16.959 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65064.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65064.log.gz 2024-09-14T06:10:16.959 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68043.log 2024-09-14T06:10:16.961 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.72331.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72331.log.gz 2024-09-14T06:10:16.961 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60942.log 2024-09-14T06:10:16.961 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.68043.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68043.log.gz 2024-09-14T06:10:16.962 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60276.log 2024-09-14T06:10:16.962 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.60942.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60942.log.gz 2024-09-14T06:10:16.963 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50126.log 2024-09-14T06:10:16.969 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.60276.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60276.log.gz 2024-09-14T06:10:16.969 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40379.log 2024-09-14T06:10:16.970 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.50126.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50126.log.gz 2024-09-14T06:10:16.971 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66108.log 2024-09-14T06:10:16.973 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.40379.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44518.log 2024-09-14T06:10:16.973 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.66108.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66108.log.gz 2024-09-14T06:10:16.973 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56962.log 2024-09-14T06:10:16.975 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.44518.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44518.log.gz 2024-09-14T06:10:16.975 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44177.log 2024-09-14T06:10:16.975 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.56962.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56962.log.gz 2024-09-14T06:10:16.977 INFO:teuthology.orchestra.run.smithi191.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.41334.log.gz 2024-09-14T06:10:16.977 INFO:teuthology.orchestra.run.smithi191.stderr:gzip 92.6% -- replaced with /var/log/ceph/cluster2.audit.log.gz 2024-09-14T06:10:16.977 INFO:teuthology.orchestra.run.smithi191.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56471.log 2024-09-14T06:10:16.977 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.44177.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56037.log 2024-09-14T06:10:16.977 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.56471.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.56471.log.gz 2024-09-14T06:10:16.978 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41506.log 2024-09-14T06:10:16.979 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.56037.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56037.log.gz 2024-09-14T06:10:16.979 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57556.log 2024-09-14T06:10:16.979 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.41506.log: 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.41506.log.gz 2024-09-14T06:10:16.980 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36759.log 2024-09-14T06:10:16.981 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.57556.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.57556.log.gz 2024-09-14T06:10:16.981 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65873.log 2024-09-14T06:10:16.983 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.36759.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53255.log 2024-09-14T06:10:16.983 INFO:teuthology.orchestra.run.smithi191.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44177.log.gz 2024-09-14T06:10:16.983 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65873.log: gzip -5 --verbose -- 88.4% /var/log/ceph/cluster2-client.mirror.47177.log 2024-09-14T06:10:16.984 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.65873.log.gz 2024-09-14T06:10:16.985 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.53255.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.36759.log.gz 2024-09-14T06:10:16.985 INFO:teuthology.orchestra.run.smithi191.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53255.log.gz 2024-09-14T06:10:16.985 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43532.log 2024-09-14T06:10:16.992 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47177.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47999.log 2024-09-14T06:10:17.002 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.43532.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62344.log 2024-09-14T06:10:17.007 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47999.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43532.log.gz 2024-09-14T06:10:17.015 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63435.log 2024-09-14T06:10:17.016 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.62344.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.62344.log.gz 2024-09-14T06:10:17.017 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67683.log 2024-09-14T06:10:17.018 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.63435.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63435.log.gz 2024-09-14T06:10:17.019 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49036.log 2024-09-14T06:10:17.019 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.67683.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67683.log.gz 2024-09-14T06:10:17.021 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47701.log 2024-09-14T06:10:17.021 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49036.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49036.log.gz 2024-09-14T06:10:17.022 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68268.log 2024-09-14T06:10:17.022 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47701.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47701.log.gz 2024-09-14T06:10:17.023 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48317.log 2024-09-14T06:10:17.024 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.68268.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68268.log.gz 2024-09-14T06:10:17.024 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41227.log 2024-09-14T06:10:17.034 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48317.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69416.log 2024-09-14T06:10:17.041 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.41227.log: 93.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43681.log 2024-09-14T06:10:17.048 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.69416.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52917.log 2024-09-14T06:10:17.049 INFO:teuthology.orchestra.run.smithi191.stderr: 88.2%/var/log/ceph/cluster1-client.mirror.43681.log: -- replaced with /var/log/ceph/cluster2-client.mirror.69416.log.gz 2024-09-14T06:10:17.053 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.41227.log.gz 2024-09-14T06:10:17.054 INFO:teuthology.orchestra.run.smithi191.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43681.log.gz 2024-09-14T06:10:17.058 INFO:teuthology.orchestra.run.smithi191.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.39221.log.gz 2024-09-14T06:10:17.067 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44598.log 2024-09-14T06:10:17.068 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.52917.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52917.log.gz 2024-09-14T06:10:17.081 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64588.log 2024-09-14T06:10:17.082 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.44598.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44598.log.gz 2024-09-14T06:10:17.101 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54581.log 2024-09-14T06:10:17.102 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64588.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64588.log.gz 2024-09-14T06:10:17.110 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66490.log 2024-09-14T06:10:17.123 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.54581.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58375.log 2024-09-14T06:10:17.124 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54581.log.gz 2024-09-14T06:10:17.124 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.66490.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66490.log.gz 2024-09-14T06:10:17.136 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61331.log 2024-09-14T06:10:17.137 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.58375.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.58375.log.gz 2024-09-14T06:10:17.145 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64406.log 2024-09-14T06:10:17.156 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.61331.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48920.log 2024-09-14T06:10:17.159 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64406.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.61331.log.gz 2024-09-14T06:10:17.172 INFO:teuthology.orchestra.run.smithi191.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64406.log.gz 2024-09-14T06:10:17.173 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.36792.log 2024-09-14T06:10:17.173 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48920.log: 80.1% -- replaced with /var/log/ceph/cluster2-client.mirror.48920.log.gz 2024-09-14T06:10:17.188 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51993.log 2024-09-14T06:10:17.201 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.2.36792.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65229.log 2024-09-14T06:10:17.201 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.51993.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51993.log.gz 2024-09-14T06:10:17.215 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70127.log 2024-09-14T06:10:17.217 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65229.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65229.log.gz 2024-09-14T06:10:17.237 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49523.log 2024-09-14T06:10:17.249 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.70127.log: 87.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54509.log 2024-09-14T06:10:17.249 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.70127.log.gz 2024-09-14T06:10:17.250 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49523.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49523.log.gz 2024-09-14T06:10:17.262 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69538.log 2024-09-14T06:10:17.262 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.54509.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.54509.log.gz 2024-09-14T06:10:17.264 INFO:teuthology.orchestra.run.smithi191.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46582.log.gz 2024-09-14T06:10:17.268 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58554.log 2024-09-14T06:10:17.269 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.69538.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.69538.log.gz 2024-09-14T06:10:17.269 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71770.log 2024-09-14T06:10:17.269 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.58554.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.58554.log.gz 2024-09-14T06:10:17.270 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62850.log 2024-09-14T06:10:17.270 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.71770.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71770.log.gz 2024-09-14T06:10:17.271 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51366.log 2024-09-14T06:10:17.272 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.62850.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62850.log.gz 2024-09-14T06:10:17.273 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58995.log 2024-09-14T06:10:17.273 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.51366.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.51366.log.gz 2024-09-14T06:10:17.285 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52290.log 2024-09-14T06:10:17.287 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.58995.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58995.log.gz 2024-09-14T06:10:17.300 INFO:teuthology.orchestra.run.smithi191.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45596.log.gz 2024-09-14T06:10:17.301 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58753.log 2024-09-14T06:10:17.301 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.52290.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52290.log.gz 2024-09-14T06:10:17.302 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49219.log 2024-09-14T06:10:17.305 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.58753.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.58753.log.gz 2024-09-14T06:10:17.305 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67300.log 2024-09-14T06:10:17.314 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49219.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52539.log 2024-09-14T06:10:17.316 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.67300.log: 86.9% 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47840.log.gz 2024-09-14T06:10:17.319 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70418.log 2024-09-14T06:10:17.323 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.52539.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.52539.log.gz 2024-09-14T06:10:17.323 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49079.log 2024-09-14T06:10:17.324 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.70418.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.70418.log.gz 2024-09-14T06:10:17.324 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53729.log 2024-09-14T06:10:17.328 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.67300.log.gz 2024-09-14T06:10:17.335 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49079.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49079.log.gz 2024-09-14T06:10:17.335 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43326.log 2024-09-14T06:10:17.338 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.53729.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.53729.log.gz 2024-09-14T06:10:17.338 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52152.log 2024-09-14T06:10:17.339 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.43326.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48317.log.gz 2024-09-14T06:10:17.340 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45939.log 2024-09-14T06:10:17.343 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.52152.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52152.log.gz 2024-09-14T06:10:17.343 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57339.log 2024-09-14T06:10:17.343 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45939.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45939.log.gz 2024-09-14T06:10:17.343 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35815.log 2024-09-14T06:10:17.345 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.57339.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57339.log.gz 2024-09-14T06:10:17.345 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37283.log 2024-09-14T06:10:17.345 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.35815.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35815.log.gz 2024-09-14T06:10:17.347 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47798.log 2024-09-14T06:10:17.347 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.37283.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63683.log 2024-09-14T06:10:17.349 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47798.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47798.log.gz 2024-09-14T06:10:17.349 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36068.log 2024-09-14T06:10:17.350 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.63683.log: 66.4% -- replaced with /var/log/ceph/cluster1-client.admin.37283.log.gz 2024-09-14T06:10:17.352 INFO:teuthology.orchestra.run.smithi191.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.63683.log.gz 2024-09-14T06:10:17.352 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45006.log 2024-09-14T06:10:17.353 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.36068.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36068.log.gz 2024-09-14T06:10:17.353 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45085.log 2024-09-14T06:10:17.354 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45006.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45006.log.gz 2024-09-14T06:10:17.354 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66400.log 2024-09-14T06:10:17.355 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45085.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45085.log.gz 2024-09-14T06:10:17.356 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49359.log 2024-09-14T06:10:17.358 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.66400.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.66400.log.gz 2024-09-14T06:10:17.358 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54465.log 2024-09-14T06:10:17.358 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49359.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.49359.log.gz 2024-09-14T06:10:17.358 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37573.log 2024-09-14T06:10:17.360 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.54465.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.54465.log.gz 2024-09-14T06:10:17.360 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45857.log 2024-09-14T06:10:17.365 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.37573.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61474.log 2024-09-14T06:10:17.365 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45857.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45857.log.gz 2024-09-14T06:10:17.365 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66130.log 2024-09-14T06:10:17.365 INFO:teuthology.orchestra.run.smithi191.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43326.log.gz 2024-09-14T06:10:17.365 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.61474.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51591.log 2024-09-14T06:10:17.365 INFO:teuthology.orchestra.run.smithi191.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.61474.log.gz 2024-09-14T06:10:17.365 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62141.log 2024-09-14T06:10:17.365 INFO:teuthology.orchestra.run.smithi191.stderr: 57.7% -- replaced with /var/log/ceph/cluster2-client.mirror.37573.log.gz 2024-09-14T06:10:17.365 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.51591.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.51591.log.gz 2024-09-14T06:10:17.368 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63786.log 2024-09-14T06:10:17.368 INFO:teuthology.orchestra.run.smithi191.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45837.log.gz 2024-09-14T06:10:17.368 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.62141.log: /var/log/ceph/cluster1-client.mirror.66130.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66130.log.gz 2024-09-14T06:10:17.368 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35477.log 2024-09-14T06:10:17.368 INFO:teuthology.orchestra.run.smithi191.stderr: 89.7%/var/log/ceph/cluster2-client.mirror.63786.log: -- replaced with /var/log/ceph/cluster2-client.mirror.62141.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45736.log 2024-09-14T06:10:17.368 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-14T06:10:17.368 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45736.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45736.log.gz 2024-09-14T06:10:17.373 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.35477.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35477.log.gz 2024-09-14T06:10:17.373 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37663.log 2024-09-14T06:10:17.373 INFO:teuthology.orchestra.run.smithi191.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63786.log.gz 2024-09-14T06:10:17.373 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47940.log 2024-09-14T06:10:17.373 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.37663.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38887.log 2024-09-14T06:10:17.373 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47940.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44091.log 2024-09-14T06:10:17.373 INFO:teuthology.orchestra.run.smithi191.stderr: 80.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47940.log.gz 2024-09-14T06:10:17.373 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.38887.log: 89.4%gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.44091.log: -- /var/log/ceph/cluster1-client.mirror.43767.log 2024-09-14T06:10:17.374 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.37663.log.gz 2024-09-14T06:10:17.374 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.43767.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47999.log.gz 2024-09-14T06:10:17.390 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46639.log 2024-09-14T06:10:17.390 INFO:teuthology.orchestra.run.smithi191.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44091.log.gz 2024-09-14T06:10:17.390 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54292.log 2024-09-14T06:10:17.390 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46639.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35816.log 2024-09-14T06:10:17.390 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46639.log.gz 2024-09-14T06:10:17.390 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.54292.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71012.log 2024-09-14T06:10:17.390 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.35816.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.54292.log.gz 2024-09-14T06:10:17.390 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35816.log.gz 2024-09-14T06:10:17.391 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66805.log 2024-09-14T06:10:17.391 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.71012.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49376.log 2024-09-14T06:10:17.391 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.66805.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61908.log 2024-09-14T06:10:17.391 INFO:teuthology.orchestra.run.smithi191.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43767.log.gz 2024-09-14T06:10:17.391 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49376.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72186.log 2024-09-14T06:10:17.391 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.61908.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.71012.log.gz 2024-09-14T06:10:17.391 INFO:teuthology.orchestra.run.smithi191.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.61908.log.gz 2024-09-14T06:10:17.391 INFO:teuthology.orchestra.run.smithi191.stderr:gzip 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66805.log.gz 2024-09-14T06:10:17.391 INFO:teuthology.orchestra.run.smithi191.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49376.log.gz 2024-09-14T06:10:17.391 INFO:teuthology.orchestra.run.smithi191.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52764.log 2024-09-14T06:10:17.391 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.72186.log: gzip 0.0% -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64919.log 2024-09-14T06:10:17.391 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.72186.log.gz 2024-09-14T06:10:17.392 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.52764.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35817.log 2024-09-14T06:10:17.392 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64919.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.52764.log.gz 2024-09-14T06:10:17.392 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50333.log 2024-09-14T06:10:17.392 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.admin.35817.log: -- /var/log/ceph/cluster1-client.mirror.55169.log 2024-09-14T06:10:17.392 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35817.log.gz 2024-09-14T06:10:17.392 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.50333.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47177.log.gz 2024-09-14T06:10:17.392 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45212.log 2024-09-14T06:10:17.392 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.55169.log: /var/log/ceph/cluster2-client.mirror.45212.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.55169.log.gz 2024-09-14T06:10:17.392 INFO:teuthology.orchestra.run.smithi191.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64919.log.gz 2024-09-14T06:10:17.392 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55579.log 2024-09-14T06:10:17.392 INFO:teuthology.orchestra.run.smithi191.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50333.log.gz 2024-09-14T06:10:17.392 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70360.log 2024-09-14T06:10:17.392 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.55579.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45998.log 2024-09-14T06:10:17.393 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4%/var/log/ceph/cluster2-client.mirror.70360.log: -- replaced with /var/log/ceph/cluster1-client.mirror.55579.log.gz 2024-09-14T06:10:17.393 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55455.log 2024-09-14T06:10:17.393 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56688.log 2024-09-14T06:10:17.393 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45998.log: /var/log/ceph/cluster2-client.mirror.55455.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47718.log 2024-09-14T06:10:17.393 INFO:teuthology.orchestra.run.smithi191.stderr: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.70360.log.gz 2024-09-14T06:10:17.393 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.56688.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55455.log.gz 2024-09-14T06:10:17.393 INFO:teuthology.orchestra.run.smithi191.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56688.log.gz 2024-09-14T06:10:17.393 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50873.log 2024-09-14T06:10:17.393 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47718.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56777.log 2024-09-14T06:10:17.393 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.50873.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42860.log 2024-09-14T06:10:17.393 INFO:teuthology.orchestra.run.smithi191.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47718.log.gz 2024-09-14T06:10:17.393 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.56777.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50873.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57638.log 2024-09-14T06:10:17.393 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-14T06:10:17.394 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.42860.log: /var/log/ceph/cluster1-client.mirror.57638.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57638.log.gz 2024-09-14T06:10:17.396 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71912.log 2024-09-14T06:10:17.397 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53323.log 2024-09-14T06:10:17.397 INFO:teuthology.orchestra.run.smithi191.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.56777.log.gz 2024-09-14T06:10:17.397 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47658.log 2024-09-14T06:10:17.397 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.53323.log: /var/log/ceph/cluster2-client.mirror.71912.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53323.log.gz 2024-09-14T06:10:17.397 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68155.log 2024-09-14T06:10:17.399 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47658.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47658.log.gz 2024-09-14T06:10:17.399 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69768.log 2024-09-14T06:10:17.399 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.68155.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68155.log.gz 2024-09-14T06:10:17.399 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71812.log 2024-09-14T06:10:17.408 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.69768.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48662.log.gz 2024-09-14T06:10:17.410 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44738.log 2024-09-14T06:10:17.411 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.71812.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.71812.log.gz 2024-09-14T06:10:17.423 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56582.log 2024-09-14T06:10:17.425 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.44738.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44738.log.gz 2024-09-14T06:10:17.431 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65043.log 2024-09-14T06:10:17.444 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48878.log 2024-09-14T06:10:17.448 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65043.log: /var/log/ceph/cluster2-client.mirror.56582.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65043.log.gz 2024-09-14T06:10:17.448 INFO:teuthology.orchestra.run.smithi191.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.56582.log.gz 2024-09-14T06:10:17.448 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47638.log 2024-09-14T06:10:17.449 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48878.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48878.log.gz 2024-09-14T06:10:17.449 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53214.log 2024-09-14T06:10:17.451 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47638.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47638.log.gz 2024-09-14T06:10:17.451 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42364.log 2024-09-14T06:10:17.451 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.53214.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.53214.log.gz 2024-09-14T06:10:17.455 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36575.log 2024-09-14T06:10:17.469 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.42364.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40552.log 2024-09-14T06:10:17.469 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.36575.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36575.log.gz 2024-09-14T06:10:17.483 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58708.log 2024-09-14T06:10:17.483 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.40552.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.40552.log.gz 2024-09-14T06:10:17.495 INFO:teuthology.orchestra.run.smithi191.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.38887.log.gz 2024-09-14T06:10:17.503 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52314.log 2024-09-14T06:10:17.504 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.58708.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.58708.log.gz 2024-09-14T06:10:17.514 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58443.log 2024-09-14T06:10:17.514 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.52314.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.52314.log.gz 2024-09-14T06:10:17.522 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45068.log 2024-09-14T06:10:17.527 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.58443.log: 93.7% 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58443.log.gz 2024-09-14T06:10:17.528 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.41731.log.gz 2024-09-14T06:10:17.534 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50441.log 2024-09-14T06:10:17.534 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45068.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45068.log.gz 2024-09-14T06:10:17.541 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63745.log 2024-09-14T06:10:17.541 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.50441.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50441.log.gz 2024-09-14T06:10:17.542 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53913.log 2024-09-14T06:10:17.542 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.63745.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.63745.log.gz 2024-09-14T06:10:17.542 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68409.log 2024-09-14T06:10:17.542 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.53913.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53913.log.gz 2024-09-14T06:10:17.542 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49641.log 2024-09-14T06:10:17.542 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.68409.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.68409.log.gz 2024-09-14T06:10:17.542 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44578.log 2024-09-14T06:10:17.542 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49641.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49641.log.gz 2024-09-14T06:10:17.542 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43747.log 2024-09-14T06:10:17.544 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.44578.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44578.log.gz 2024-09-14T06:10:17.544 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69866.log 2024-09-14T06:10:17.549 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.43747.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43747.log.gz 2024-09-14T06:10:17.550 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50670.log 2024-09-14T06:10:17.552 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.69866.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69866.log.gz 2024-09-14T06:10:17.552 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65811.log 2024-09-14T06:10:17.553 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.50670.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50670.log.gz 2024-09-14T06:10:17.554 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38726.log 2024-09-14T06:10:17.556 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65811.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65811.log.gz 2024-09-14T06:10:17.556 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39033.log 2024-09-14T06:10:17.556 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.38726.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.59637.log 2024-09-14T06:10:17.560 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.39033.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.38726.log.gz 2024-09-14T06:10:17.561 INFO:teuthology.orchestra.run.smithi191.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.39033.log.gz 2024-09-14T06:10:17.572 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56757.log 2024-09-14T06:10:17.574 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.2.59637.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.2.59637.log.gz 2024-09-14T06:10:17.585 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58226.log 2024-09-14T06:10:17.587 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.56757.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56757.log.gz 2024-09-14T06:10:17.596 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54268.log 2024-09-14T06:10:17.596 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.58226.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58226.log.gz 2024-09-14T06:10:17.602 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54340.log 2024-09-14T06:10:17.615 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.54268.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54268.log.gz 2024-09-14T06:10:17.627 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48900.log 2024-09-14T06:10:17.627 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.54340.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.54340.log.gz 2024-09-14T06:10:17.636 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.36914.log 2024-09-14T06:10:17.652 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48900.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44326.log 2024-09-14T06:10:17.659 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.2.36914.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.69261.log 2024-09-14T06:10:17.665 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.44326.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.2.36914.log.gz 2024-09-14T06:10:17.679 INFO:teuthology.orchestra.run.smithi191.stderr: 93.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62414.log 2024-09-14T06:10:17.679 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.44326.log.gz 2024-09-14T06:10:17.681 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.0.69261.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45212.log.gz 2024-09-14T06:10:17.683 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71098.log 2024-09-14T06:10:17.683 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.62414.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62414.log.gz 2024-09-14T06:10:17.688 INFO:teuthology.orchestra.run.smithi191.stderr: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.0.69261.log.gz 2024-09-14T06:10:17.694 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61775.log 2024-09-14T06:10:17.695 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.71098.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.71098.log.gz 2024-09-14T06:10:17.709 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68907.log 2024-09-14T06:10:17.710 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.61775.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.61775.log.gz 2024-09-14T06:10:17.722 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48799.log 2024-09-14T06:10:17.723 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.68907.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.68907.log.gz 2024-09-14T06:10:17.738 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44618.log 2024-09-14T06:10:17.739 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48799.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48799.log.gz 2024-09-14T06:10:17.754 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62257.log 2024-09-14T06:10:17.755 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.44618.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44618.log.gz 2024-09-14T06:10:17.769 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49116.log 2024-09-14T06:10:17.769 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.62257.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62257.log.gz 2024-09-14T06:10:17.777 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53391.log 2024-09-14T06:10:17.786 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49116.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73011.log 2024-09-14T06:10:17.796 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.53391.log: 87.4% 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49116.log.gz 2024-09-14T06:10:17.796 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67570.log 2024-09-14T06:10:17.799 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.73011.log: -- replaced with /var/log/ceph/cluster1-client.mirror.53391.log.gz 2024-09-14T06:10:17.808 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.73011.log.gz 2024-09-14T06:10:17.813 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68828.log 2024-09-14T06:10:17.813 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.67570.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67570.log.gz 2024-09-14T06:10:17.820 INFO:teuthology.orchestra.run.smithi191.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.42364.log.gz 2024-09-14T06:10:17.820 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70477.log 2024-09-14T06:10:17.821 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.68828.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.68828.log.gz 2024-09-14T06:10:17.821 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48434.log 2024-09-14T06:10:17.822 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.70477.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.70477.log.gz 2024-09-14T06:10:17.822 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59811.log 2024-09-14T06:10:17.823 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48434.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48434.log.gz 2024-09-14T06:10:17.824 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65852.log 2024-09-14T06:10:17.824 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.59811.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59811.log.gz 2024-09-14T06:10:17.825 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55497.log 2024-09-14T06:10:17.827 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65852.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65852.log.gz 2024-09-14T06:10:17.827 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46143.log 2024-09-14T06:10:17.827 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.55497.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.55497.log.gz 2024-09-14T06:10:17.828 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71870.log 2024-09-14T06:10:17.828 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46143.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46143.log.gz 2024-09-14T06:10:17.829 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55214.log 2024-09-14T06:10:17.830 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.71870.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.71870.log.gz 2024-09-14T06:10:17.831 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43263.log 2024-09-14T06:10:17.832 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.55214.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55214.log.gz 2024-09-14T06:10:17.833 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64483.log 2024-09-14T06:10:17.838 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.43263.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43263.log.gz 2024-09-14T06:10:17.839 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55097.log 2024-09-14T06:10:17.847 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64483.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64483.log.gz 2024-09-14T06:10:17.848 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58974.log 2024-09-14T06:10:17.848 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.55097.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55097.log.gz 2024-09-14T06:10:17.850 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45105.log 2024-09-14T06:10:17.851 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.58974.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.58974.log.gz 2024-09-14T06:10:17.851 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40624.log 2024-09-14T06:10:17.851 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45105.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45105.log.gz 2024-09-14T06:10:17.851 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67953.log 2024-09-14T06:10:17.853 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.40624.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.40624.log.gz 2024-09-14T06:10:17.853 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37445.log 2024-09-14T06:10:17.854 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.67953.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67953.log.gz 2024-09-14T06:10:17.854 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70521.log 2024-09-14T06:10:17.855 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.37445.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37445.log.gz 2024-09-14T06:10:17.855 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65353.log 2024-09-14T06:10:17.857 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.70521.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.70521.log.gz 2024-09-14T06:10:17.857 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64546.log 2024-09-14T06:10:17.860 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65353.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65353.log.gz 2024-09-14T06:10:17.860 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37368.log 2024-09-14T06:10:17.863 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64546.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64546.log.gz 2024-09-14T06:10:17.864 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.70558.log 2024-09-14T06:10:17.864 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.37368.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71657.log 2024-09-14T06:10:17.871 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.2.70558.log: 57.7% -- replaced with /var/log/ceph/cluster1-client.mirror.37368.log.gz 2024-09-14T06:10:17.871 INFO:teuthology.orchestra.run.smithi191.stderr: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.2.70558.log.gz 2024-09-14T06:10:17.872 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39996.log 2024-09-14T06:10:17.872 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.71657.log: 76.3% -- replaced with /var/log/ceph/cluster2-client.mirror.71657.log.gz 2024-09-14T06:10:17.872 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64055.log 2024-09-14T06:10:17.873 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.39996.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58419.log 2024-09-14T06:10:17.875 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64055.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64055.log.gz 2024-09-14T06:10:17.875 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56640.log 2024-09-14T06:10:17.876 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.58419.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.58419.log.gz 2024-09-14T06:10:17.877 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60824.log 2024-09-14T06:10:17.881 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.56640.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56640.log.gz 2024-09-14T06:10:17.881 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46702.log 2024-09-14T06:10:17.883 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.60824.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.60824.log.gz 2024-09-14T06:10:17.883 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59953.log 2024-09-14T06:10:17.885 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46702.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46702.log.gz 2024-09-14T06:10:17.885 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59637.log 2024-09-14T06:10:17.886 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.59953.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59953.log.gz 2024-09-14T06:10:17.886 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49058.log 2024-09-14T06:10:17.891 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.59637.log: 77.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59637.log.gz 2024-09-14T06:10:17.891 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45270.log 2024-09-14T06:10:17.910 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49058.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44989.log 2024-09-14T06:10:17.910 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45270.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45270.log.gz 2024-09-14T06:10:17.918 INFO:teuthology.orchestra.run.smithi191.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.71912.log.gz 2024-09-14T06:10:17.919 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61127.log 2024-09-14T06:10:17.919 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.44989.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44989.log.gz 2024-09-14T06:10:17.920 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41312.log 2024-09-14T06:10:17.921 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.61127.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61127.log.gz 2024-09-14T06:10:17.921 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39673.log 2024-09-14T06:10:17.922 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.41312.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.41312.log.gz 2024-09-14T06:10:17.923 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.48514.log 2024-09-14T06:10:17.923 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.39673.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56013.log 2024-09-14T06:10:17.929 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.1.48514.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.39673.log.gz 2024-09-14T06:10:17.930 INFO:teuthology.orchestra.run.smithi191.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.1.48514.log.gz 2024-09-14T06:10:17.931 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65436.log 2024-09-14T06:10:17.933 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.56013.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.56013.log.gz 2024-09-14T06:10:17.933 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.0.log 2024-09-14T06:10:17.934 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65436.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65436.log.gz 2024-09-14T06:10:17.934 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57837.log 2024-09-14T06:10:17.936 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-osd.0.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.42860.log.gz 2024-09-14T06:10:17.939 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.57837.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57837.log.gz 2024-09-14T06:10:17.939 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56495.log 2024-09-14T06:10:17.939 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43170.log 2024-09-14T06:10:17.942 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.56495.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56495.log.gz 2024-09-14T06:10:17.943 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64179.log 2024-09-14T06:10:17.943 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.43170.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.43170.log.gz 2024-09-14T06:10:17.943 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62685.log 2024-09-14T06:10:17.943 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66895.log 2024-09-14T06:10:17.943 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.62685.log: /var/log/ceph/cluster2-client.mirror.64179.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.62685.log.gz 2024-09-14T06:10:17.943 INFO:teuthology.orchestra.run.smithi191.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64179.log.gz 2024-09-14T06:10:17.944 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51228.log 2024-09-14T06:10:17.944 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.66895.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66895.log.gz 2024-09-14T06:10:17.944 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42530.log 2024-09-14T06:10:17.945 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.51228.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51228.log.gz 2024-09-14T06:10:17.945 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69261.log 2024-09-14T06:10:17.947 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60001.log 2024-09-14T06:10:17.947 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.69261.log: /var/log/ceph/cluster1-client.mirror.42530.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.69261.log.gz 2024-09-14T06:10:17.947 INFO:teuthology.orchestra.run.smithi191.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42530.log.gz 2024-09-14T06:10:17.947 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39149.log 2024-09-14T06:10:17.949 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63890.log 2024-09-14T06:10:17.949 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.39149.log: /var/log/ceph/cluster1-client.mirror.60001.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.60001.log.gz 2024-09-14T06:10:17.949 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59215.log 2024-09-14T06:10:17.951 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.63890.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63890.log.gz 2024-09-14T06:10:17.951 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46185.log 2024-09-14T06:10:17.951 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.59215.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59215.log.gz 2024-09-14T06:10:17.952 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47030.log 2024-09-14T06:10:17.955 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46185.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.39149.log.gz 2024-09-14T06:10:17.959 INFO:teuthology.orchestra.run.smithi191.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.69768.log.gz 2024-09-14T06:10:17.963 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71747.log 2024-09-14T06:10:17.968 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47030.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45675.log 2024-09-14T06:10:17.968 INFO:teuthology.orchestra.run.smithi191.stderr: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47030.log.gz 2024-09-14T06:10:17.968 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.71747.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57059.log 2024-09-14T06:10:17.969 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45675.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71747.log.gz 2024-09-14T06:10:17.969 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50762.log 2024-09-14T06:10:17.969 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.57059.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.57059.log.gz 2024-09-14T06:10:17.969 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67278.log 2024-09-14T06:10:17.969 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.50762.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.50762.log.gz 2024-09-14T06:10:17.969 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71390.log 2024-09-14T06:10:17.974 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.67278.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67278.log.gz 2024-09-14T06:10:17.974 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36197.log 2024-09-14T06:10:17.974 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.71390.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40292.log 2024-09-14T06:10:17.974 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.36197.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71390.log.gz 2024-09-14T06:10:17.974 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36197.log.gz 2024-09-14T06:10:17.974 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45190.log 2024-09-14T06:10:17.974 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.40292.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43638.log 2024-09-14T06:10:17.974 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48741.log 2024-09-14T06:10:17.974 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45190.log: /var/log/ceph/cluster1-client.mirror.43638.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45190.log.gz 2024-09-14T06:10:17.974 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42486.log 2024-09-14T06:10:17.976 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48741.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45998.log.gz 2024-09-14T06:10:17.977 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65188.log 2024-09-14T06:10:17.979 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.42486.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.42486.log.gz 2024-09-14T06:10:17.979 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59977.log 2024-09-14T06:10:17.981 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65188.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65188.log.gz 2024-09-14T06:10:17.981 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43575.log 2024-09-14T06:10:17.982 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.59977.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59977.log.gz 2024-09-14T06:10:17.982 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59075.log 2024-09-14T06:10:17.986 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.43575.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49219.log.gz 2024-09-14T06:10:17.987 INFO:teuthology.orchestra.run.smithi191.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.2.36792.log.gz 2024-09-14T06:10:17.987 INFO:teuthology.orchestra.run.smithi191.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43575.log.gz 2024-09-14T06:10:17.988 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57961.log 2024-09-14T06:10:17.989 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.59075.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.59075.log.gz 2024-09-14T06:10:17.991 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43939.log 2024-09-14T06:10:17.991 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44718.log 2024-09-14T06:10:17.991 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.57961.log: /var/log/ceph/cluster1-client.mirror.43939.log: 93.3% 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.57961.log.gz 2024-09-14T06:10:17.991 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.43638.log.gz 2024-09-14T06:10:17.991 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42708.log 2024-09-14T06:10:17.992 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46381.log 2024-09-14T06:10:17.992 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.44718.log: /var/log/ceph/cluster1-client.mirror.42708.log: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.42708.log.gz 2024-09-14T06:10:17.992 INFO:teuthology.orchestra.run.smithi191.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44718.log.gz 2024-09-14T06:10:17.993 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46502.log 2024-09-14T06:10:17.994 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46381.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46381.log.gz 2024-09-14T06:10:17.994 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68372.log 2024-09-14T06:10:18.002 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.mirror.46502.log: /var/log/ceph/cluster1-client.mirror.56278.log 2024-09-14T06:10:18.002 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.68372.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.68372.log.gz 2024-09-14T06:10:18.002 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41554.log 2024-09-14T06:10:18.002 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.56278.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.56278.log.gz 2024-09-14T06:10:18.002 INFO:teuthology.orchestra.run.smithi191.stderr: 93.4%gzip -- replaced with /var/log/ceph/cluster1-client.mirror.43939.log.gz -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39559.log 2024-09-14T06:10:18.002 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-14T06:10:18.002 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.41554.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41554.log.gz 2024-09-14T06:10:18.003 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42506.log 2024-09-14T06:10:18.003 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.39559.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64899.log 2024-09-14T06:10:18.003 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72593.log 2024-09-14T06:10:18.003 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.42506.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.42506.log.gz 2024-09-14T06:10:18.003 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41667.log 2024-09-14T06:10:18.003 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.72593.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72593.log.gz 2024-09-14T06:10:18.003 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43146.log 2024-09-14T06:10:18.005 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.41667.log: /var/log/ceph/cluster2-client.mirror.64899.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64899.log.gz 2024-09-14T06:10:18.006 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43618.log 2024-09-14T06:10:18.006 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.43146.log: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.43146.log.gz 2024-09-14T06:10:18.006 INFO:teuthology.orchestra.run.smithi191.stderr: 81.6% -- replaced with /var/log/ceph/cluster1-client.mirror.39996.log.gz 2024-09-14T06:10:18.006 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45250.log 2024-09-14T06:10:18.009 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.43618.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41667.log.gz 2024-09-14T06:10:18.009 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51543.log 2024-09-14T06:10:18.011 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45250.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45250.log.gz 2024-09-14T06:10:18.011 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67593.log 2024-09-14T06:10:18.012 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.51543.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43618.log.gz 2024-09-14T06:10:18.012 INFO:teuthology.orchestra.run.smithi191.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51543.log.gz 2024-09-14T06:10:18.017 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48374.log 2024-09-14T06:10:18.017 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.67593.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70030.log 2024-09-14T06:10:18.017 INFO:teuthology.orchestra.run.smithi191.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67593.log.gz 2024-09-14T06:10:18.017 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48374.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36914.log 2024-09-14T06:10:18.017 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.70030.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48374.log.gz 2024-09-14T06:10:18.017 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.70030.log.gz 2024-09-14T06:10:18.017 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55627.log 2024-09-14T06:10:18.017 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.36914.log: -- /var/log/ceph/cluster2-client.mirror.56206.log 2024-09-14T06:10:18.018 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47781.log 2024-09-14T06:10:18.018 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.55627.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55627.log.gz 2024-09-14T06:10:18.018 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63351.log 2024-09-14T06:10:18.022 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47781.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47781.log.gz 2024-09-14T06:10:18.022 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50987.log 2024-09-14T06:10:18.022 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.63351.log: /var/log/ceph/cluster2-client.mirror.56206.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56206.log.gz 2024-09-14T06:10:18.022 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57514.log 2024-09-14T06:10:18.022 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.50987.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.50987.log.gz 2024-09-14T06:10:18.022 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40436.log 2024-09-14T06:10:18.022 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.57514.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63351.log.gz 2024-09-14T06:10:18.023 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70813.log 2024-09-14T06:10:18.024 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.40436.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.40436.log.gz 2024-09-14T06:10:18.024 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60689.log 2024-09-14T06:10:18.032 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.70813.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57514.log.gz 2024-09-14T06:10:18.032 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71242.log 2024-09-14T06:10:18.032 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.60689.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60689.log.gz 2024-09-14T06:10:18.036 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44941.log 2024-09-14T06:10:18.042 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.71242.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.71242.log.gz 2024-09-14T06:10:18.042 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66940.log 2024-09-14T06:10:18.042 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.44941.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44941.log.gz 2024-09-14T06:10:18.042 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49138.log 2024-09-14T06:10:18.042 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.66940.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66940.log.gz 2024-09-14T06:10:18.042 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52515.log 2024-09-14T06:10:18.055 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49138.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52785.log 2024-09-14T06:10:18.056 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.52515.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52515.log.gz 2024-09-14T06:10:18.062 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49498.log 2024-09-14T06:10:18.065 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.52785.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48900.log.gz 2024-09-14T06:10:18.065 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64651.log 2024-09-14T06:10:18.073 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49498.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.38951.log.gz 2024-09-14T06:10:18.077 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39786.log 2024-09-14T06:10:18.078 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64651.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64651.log.gz 2024-09-14T06:10:18.089 INFO:teuthology.orchestra.run.smithi191.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.40292.log.gz 2024-09-14T06:10:18.090 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55193.log 2024-09-14T06:10:18.093 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.39786.log: 80.8% -- replaced with /var/log/ceph/cluster1-client.mirror.36914.log.gz 2024-09-14T06:10:18.101 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60621.log 2024-09-14T06:10:18.101 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.55193.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.55193.log.gz 2024-09-14T06:10:18.104 INFO:teuthology.orchestra.run.smithi191.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.39559.log.gz 2024-09-14T06:10:18.110 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68200.log 2024-09-14T06:10:18.124 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.60621.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60621.log.gz 2024-09-14T06:10:18.127 INFO:teuthology.orchestra.run.smithi191.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.39786.log.gz 2024-09-14T06:10:18.141 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70275.log 2024-09-14T06:10:18.141 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.68200.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68200.log.gz 2024-09-14T06:10:18.150 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59929.log 2024-09-14T06:10:18.159 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.70275.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44283.log 2024-09-14T06:10:18.159 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.59929.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59929.log.gz 2024-09-14T06:10:18.163 INFO:teuthology.orchestra.run.smithi191.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70275.log.gz 2024-09-14T06:10:18.168 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64837.log 2024-09-14T06:10:18.168 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.44283.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52785.log.gz 2024-09-14T06:10:18.177 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54928.log 2024-09-14T06:10:18.186 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64837.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44283.log.gz 2024-09-14T06:10:18.190 INFO:teuthology.orchestra.run.smithi191.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64837.log.gz 2024-09-14T06:10:18.192 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43102.log 2024-09-14T06:10:18.193 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.54928.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54928.log.gz 2024-09-14T06:10:18.201 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42897.log 2024-09-14T06:10:18.215 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.43102.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.43102.log.gz 2024-09-14T06:10:18.218 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47234.log 2024-09-14T06:10:18.219 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.42897.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.42897.log.gz 2024-09-14T06:10:18.231 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45758.log 2024-09-14T06:10:18.232 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47234.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47234.log.gz 2024-09-14T06:10:18.242 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72211.log 2024-09-14T06:10:18.251 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45758.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61035.log 2024-09-14T06:10:18.251 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.72211.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72211.log.gz 2024-09-14T06:10:18.260 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60064.log 2024-09-14T06:10:18.275 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.61035.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61035.log.gz 2024-09-14T06:10:18.278 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59191.log 2024-09-14T06:10:18.279 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.60064.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.60064.log.gz 2024-09-14T06:10:18.300 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55931.log 2024-09-14T06:10:18.301 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.59191.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59191.log.gz 2024-09-14T06:10:18.313 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67008.log 2024-09-14T06:10:18.322 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.55931.log: 89.7% 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46502.log.gz 2024-09-14T06:10:18.326 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.55931.log.gz 2024-09-14T06:10:18.335 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48020.log 2024-09-14T06:10:18.335 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.67008.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67008.log.gz 2024-09-14T06:10:18.345 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57594.log 2024-09-14T06:10:18.346 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48020.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48020.log.gz 2024-09-14T06:10:18.350 INFO:teuthology.orchestra.run.smithi191.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48741.log.gz 2024-09-14T06:10:18.350 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47957.log 2024-09-14T06:10:18.352 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.57594.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57594.log.gz 2024-09-14T06:10:18.352 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52827.log 2024-09-14T06:10:18.354 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47957.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47957.log.gz 2024-09-14T06:10:18.354 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48699.log 2024-09-14T06:10:18.355 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.52827.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52827.log.gz 2024-09-14T06:10:18.355 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62767.log 2024-09-14T06:10:18.357 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48699.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48699.log.gz 2024-09-14T06:10:18.357 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48196.log 2024-09-14T06:10:18.361 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.62767.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.62767.log.gz 2024-09-14T06:10:18.361 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48841.log 2024-09-14T06:10:18.363 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48196.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48196.log.gz 2024-09-14T06:10:18.363 INFO:teuthology.orchestra.run.smithi191.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49058.log.gz 2024-09-14T06:10:18.363 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40272.log 2024-09-14T06:10:18.363 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48841.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48841.log.gz 2024-09-14T06:10:18.368 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46602.log 2024-09-14T06:10:18.368 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.40272.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67120.log 2024-09-14T06:10:18.368 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46602.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47382.log 2024-09-14T06:10:18.368 INFO:teuthology.orchestra.run.smithi191.stderr: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46602.log.gz 2024-09-14T06:10:18.368 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.67120.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63642.log 2024-09-14T06:10:18.368 INFO:teuthology.orchestra.run.smithi191.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67120.log.gz 2024-09-14T06:10:18.368 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47382.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71118.log 2024-09-14T06:10:18.368 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.63642.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.47382.log.gz 2024-09-14T06:10:18.368 INFO:teuthology.orchestra.run.smithi191.stderr: 88.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67165.log 2024-09-14T06:10:18.369 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.63642.log.gz 2024-09-14T06:10:18.369 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44197.log 2024-09-14T06:10:18.369 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.71118.log: /var/log/ceph/cluster1-client.mirror.67165.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67165.log.gz 2024-09-14T06:10:18.369 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69395.log 2024-09-14T06:10:18.371 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.44197.log: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.71118.log.gz 2024-09-14T06:10:18.371 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54706.log 2024-09-14T06:10:18.371 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.69395.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.69395.log.gz 2024-09-14T06:10:18.371 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58512.log 2024-09-14T06:10:18.373 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.54706.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.54706.log.gz 2024-09-14T06:10:18.373 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48337.log 2024-09-14T06:10:18.373 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.58512.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44197.log.gz 2024-09-14T06:10:18.374 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69709.log 2024-09-14T06:10:18.374 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48337.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48337.log.gz 2024-09-14T06:10:18.377 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53142.log 2024-09-14T06:10:18.377 INFO:teuthology.orchestra.run.smithi191.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58512.log.gz 2024-09-14T06:10:18.377 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.69709.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.69709.log.gz 2024-09-14T06:10:18.377 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35941.log 2024-09-14T06:10:18.377 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43810.log 2024-09-14T06:10:18.377 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.53142.log: /var/log/ceph/cluster2-client.admin.35941.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35941.log.gz 2024-09-14T06:10:18.378 INFO:teuthology.orchestra.run.smithi191.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53142.log.gz 2024-09-14T06:10:18.378 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64961.log 2024-09-14T06:10:18.380 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.43810.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57421.log 2024-09-14T06:10:18.380 INFO:teuthology.orchestra.run.smithi191.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46185.log.gz 2024-09-14T06:10:18.381 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64961.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64961.log.gz 2024-09-14T06:10:18.382 INFO:teuthology.orchestra.run.smithi191.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43810.log.gz 2024-09-14T06:10:18.388 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.53954.log 2024-09-14T06:10:18.388 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.57421.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50690.log 2024-09-14T06:10:18.388 INFO:teuthology.orchestra.run.smithi191.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.57421.log.gz 2024-09-14T06:10:18.388 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.2.53954.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66355.log 2024-09-14T06:10:18.388 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68313.log 2024-09-14T06:10:18.388 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.50690.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50690.log.gz 2024-09-14T06:10:18.388 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66985.log 2024-09-14T06:10:18.388 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.68313.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68313.log.gz 2024-09-14T06:10:18.389 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43218.log 2024-09-14T06:10:18.389 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.66985.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66985.log.gz 2024-09-14T06:10:18.389 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71056.log 2024-09-14T06:10:18.391 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.43218.log: 89.3%/var/log/ceph/cluster1-client.mirror.66355.log: -- replaced with /var/log/ceph/cluster1-client.mirror.43218.log.gz 2024-09-14T06:10:18.391 INFO:teuthology.orchestra.run.smithi191.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66355.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51680.log 2024-09-14T06:10:18.391 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-14T06:10:18.391 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.71056.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.71056.log.gz 2024-09-14T06:10:18.391 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51035.log 2024-09-14T06:10:18.395 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.51680.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.51680.log.gz 2024-09-14T06:10:18.395 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38746.log 2024-09-14T06:10:18.396 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36029.log 2024-09-14T06:10:18.396 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.38746.log: /var/log/ceph/cluster1-client.mirror.51035.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59408.log 2024-09-14T06:10:18.396 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4%/var/log/ceph/cluster2-client.admin.36029.log: -- replaced with /var/log/ceph/cluster1-client.mirror.51035.log.gz 2024-09-14T06:10:18.396 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.49270.log 2024-09-14T06:10:18.396 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.59408.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59408.log.gz 2024-09-14T06:10:18.396 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44154.log 2024-09-14T06:10:18.399 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.1.49270.log: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.2.53954.log.gz 2024-09-14T06:10:18.399 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36029.log.gz 2024-09-14T06:10:18.399 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.38746.log.gz 2024-09-14T06:10:18.399 INFO:teuthology.orchestra.run.smithi191.stderr: /var/log/ceph/cluster1-client.mirror.60549.log 2024-09-14T06:10:18.399 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.44154.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52692.log 2024-09-14T06:10:18.399 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.60549.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.1.49270.log.gz 2024-09-14T06:10:18.403 INFO:teuthology.orchestra.run.smithi191.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.60549.log.gz 2024-09-14T06:10:18.403 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61838.log 2024-09-14T06:10:18.403 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.52692.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43790.log 2024-09-14T06:10:18.403 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.61838.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52692.log.gz 2024-09-14T06:10:18.403 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63600.log 2024-09-14T06:10:18.403 INFO:teuthology.orchestra.run.smithi191.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.61838.log.gz 2024-09-14T06:10:18.404 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.43790.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60022.log 2024-09-14T06:10:18.405 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.63600.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44154.log.gz 2024-09-14T06:10:18.405 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-mon.a.log 2024-09-14T06:10:18.407 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.60022.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60022.log.gz 2024-09-14T06:10:18.407 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44048.log 2024-09-14T06:10:18.407 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66017.log 2024-09-14T06:10:18.409 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.44048.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43790.log.gz 2024-09-14T06:10:18.409 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69746.log 2024-09-14T06:10:18.412 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.66017.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63600.log.gz 2024-09-14T06:10:18.412 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49604.log 2024-09-14T06:10:18.412 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.69746.log: 87.5% 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.66017.log.gz 2024-09-14T06:10:18.413 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.69746.log.gz 2024-09-14T06:10:18.413 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40689.log 2024-09-14T06:10:18.418 INFO:teuthology.orchestra.run.smithi191.stderr: 93.4%gzip -- replaced with /var/log/ceph/cluster2-client.mirror.44048.log.gz -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42207.log 2024-09-14T06:10:18.418 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-14T06:10:18.419 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49604.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65395.log 2024-09-14T06:10:18.419 INFO:teuthology.orchestra.run.smithi191.stderr: 80.4%/var/log/ceph/cluster2-client.mirror.42207.log: -- replaced with /var/log/ceph/cluster2-client.mirror.49604.log.gz 2024-09-14T06:10:18.419 INFO:teuthology.orchestra.run.smithi191.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.42207.log.gz 2024-09-14T06:10:18.419 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45378.log/var/log/ceph/cluster1-client.mirror.40689.log: 2024-09-14T06:10:18.419 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65395.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65395.log.gz 2024-09-14T06:10:18.419 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56230.log 2024-09-14T06:10:18.419 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45378.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45378.log.gz 2024-09-14T06:10:18.419 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42081.log 2024-09-14T06:10:18.419 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.56230.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55820.log 2024-09-14T06:10:18.419 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.42081.log: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.42081.log.gz 2024-09-14T06:10:18.419 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60761.log 2024-09-14T06:10:18.426 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.55820.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55820.log.gz 2024-09-14T06:10:18.426 INFO:teuthology.orchestra.run.smithi191.stderr: 93.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63393.log 2024-09-14T06:10:18.426 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.40689.log.gz 2024-09-14T06:10:18.426 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39105.log 2024-09-14T06:10:18.426 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.60761.log: 87.7%/var/log/ceph/cluster2-client.mirror.63393.log: -- replaced with /var/log/ceph/cluster1-client.mirror.56230.log.gz 2024-09-14T06:10:18.426 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- 88.3% /var/log/ceph/cluster1-client.mirror.55692.log 2024-09-14T06:10:18.426 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.63393.log.gz 2024-09-14T06:10:18.426 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51615.log 2024-09-14T06:10:18.426 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71853.log 2024-09-14T06:10:18.427 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71496.log 2024-09-14T06:10:18.427 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.39105.log: /var/log/ceph/cluster1-client.mirror.55692.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.60761.log.gz 2024-09-14T06:10:18.427 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.51615.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.51615.log.gz 2024-09-14T06:10:18.427 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62642.log 2024-09-14T06:10:18.427 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.71496.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.71496.log.gz 2024-09-14T06:10:18.427 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.71853.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55692.log.gz 2024-09-14T06:10:18.427 INFO:teuthology.orchestra.run.smithi191.stderr: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.71853.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64878.log 2024-09-14T06:10:18.427 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.62642.log: 2024-09-14T06:10:18.427 INFO:teuthology.orchestra.run.smithi191.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.39105.log.gz 2024-09-14T06:10:18.428 INFO:teuthology.orchestra.run.smithi191.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.62642.log.gz 2024-09-14T06:10:18.428 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39721.log 2024-09-14T06:10:18.428 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59881.log 2024-09-14T06:10:18.429 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64878.log: /var/log/ceph/cluster1-client.mirror.39721.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64878.log.gz 2024-09-14T06:10:18.432 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46480.log 2024-09-14T06:10:18.432 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.59881.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52443.log 2024-09-14T06:10:18.433 INFO:teuthology.orchestra.run.smithi191.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59881.log.gz 2024-09-14T06:10:18.433 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46480.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43394.log 2024-09-14T06:10:18.433 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.52443.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46480.log.gz 2024-09-14T06:10:18.433 INFO:teuthology.orchestra.run.smithi191.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52443.log.gz 2024-09-14T06:10:18.433 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61635.log 2024-09-14T06:10:18.436 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.43394.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.43394.log.gz 2024-09-14T06:10:18.436 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.46037.log 2024-09-14T06:10:18.436 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.61635.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.61635.log.gz 2024-09-14T06:10:18.436 INFO:teuthology.orchestra.run.smithi191.stderr: 88.2%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64344.log 2024-09-14T06:10:18.437 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.39721.log.gz 2024-09-14T06:10:18.437 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.0.46037.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65085.log 2024-09-14T06:10:18.437 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64344.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.47293.log 2024-09-14T06:10:18.439 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65085.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65085.log.gz 2024-09-14T06:10:18.439 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57879.log 2024-09-14T06:10:18.439 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.0.47293.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64344.log.gz 2024-09-14T06:10:18.439 INFO:teuthology.orchestra.run.smithi191.stderr: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.0.46037.log.gz 2024-09-14T06:10:18.439 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50375.log 2024-09-14T06:10:18.439 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.57879.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.57879.log.gz 2024-09-14T06:10:18.440 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58134.log 2024-09-14T06:10:18.442 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.50375.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56560.log 2024-09-14T06:10:18.442 INFO:teuthology.orchestra.run.smithi191.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50375.log.gz 2024-09-14T06:10:18.442 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.58134.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41182.log 2024-09-14T06:10:18.443 INFO:teuthology.orchestra.run.smithi191.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58134.log.gz 2024-09-14T06:10:18.444 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.41182.log: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.41182.log.gz 2024-09-14T06:10:18.444 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.56560.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.0.47293.log.gz 2024-09-14T06:10:18.445 INFO:teuthology.orchestra.run.smithi191.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56560.log.gz 2024-09-14T06:10:18.446 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40131.log 2024-09-14T06:10:18.451 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60412.log 2024-09-14T06:10:18.451 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.40131.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36617.log 2024-09-14T06:10:18.451 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58925.log 2024-09-14T06:10:18.452 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.60412.log: /var/log/ceph/cluster2-client.admin.36617.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36617.log.gz 2024-09-14T06:10:18.452 INFO:teuthology.orchestra.run.smithi191.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.60412.log.gz 2024-09-14T06:10:18.452 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66535.log 2024-09-14T06:10:18.452 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.58925.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43704.log 2024-09-14T06:10:18.452 INFO:teuthology.orchestra.run.smithi191.stderr: 87.3%/var/log/ceph/cluster1-client.mirror.66535.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.40131.log.gz 2024-09-14T06:10:18.452 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.58925.log.gz 2024-09-14T06:10:18.452 INFO:teuthology.orchestra.run.smithi191.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66535.log.gz 2024-09-14T06:10:18.452 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51567.log 2024-09-14T06:10:18.452 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.43704.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48417.log 2024-09-14T06:10:18.465 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.51567.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71300.log 2024-09-14T06:10:18.466 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48417.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48417.log.gz 2024-09-14T06:10:18.466 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.1.log 2024-09-14T06:10:18.466 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.71300.log: 84.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71300.log.gz 2024-09-14T06:10:18.466 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69931.log 2024-09-14T06:10:18.466 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-osd.1.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.51567.log.gz 2024-09-14T06:10:18.466 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52580.log 2024-09-14T06:10:18.466 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.69931.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66064.log 2024-09-14T06:10:18.466 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.52580.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.69931.log.gz 2024-09-14T06:10:18.466 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65312.log 2024-09-14T06:10:18.466 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.66064.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.66064.log.gz 2024-09-14T06:10:18.466 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53460.log 2024-09-14T06:10:18.466 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65312.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43704.log.gz 2024-09-14T06:10:18.466 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68927.log 2024-09-14T06:10:18.467 INFO:teuthology.orchestra.run.smithi191.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65312.log.gz 2024-09-14T06:10:18.467 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.53460.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61450.log 2024-09-14T06:10:18.467 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.68927.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68927.log.gz 2024-09-14T06:10:18.467 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47818.log 2024-09-14T06:10:18.467 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.61450.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61450.log.gz 2024-09-14T06:10:18.467 INFO:teuthology.orchestra.run.smithi191.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.53460.log.gz 2024-09-14T06:10:18.467 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70558.log 2024-09-14T06:10:18.467 INFO:teuthology.orchestra.run.smithi191.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52580.log.gz 2024-09-14T06:10:18.467 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.mirror.47818.log: /var/log/ceph/cluster1-client.mirror.43724.log 2024-09-14T06:10:18.467 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.70558.log: gzip 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47818.log.gz 2024-09-14T06:10:18.467 INFO:teuthology.orchestra.run.smithi191.stderr: 80.4% -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53709.log 2024-09-14T06:10:18.467 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.70558.log.gz 2024-09-14T06:10:18.467 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37539.log 2024-09-14T06:10:18.468 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.43724.log: /var/log/ceph/cluster2-client.mirror.53709.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70733.log 2024-09-14T06:10:18.468 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.37539.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43306.log 2024-09-14T06:10:18.468 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.70733.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.53709.log.gz 2024-09-14T06:10:18.468 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50015.log 2024-09-14T06:10:18.468 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.43306.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.70733.log.gz 2024-09-14T06:10:18.469 INFO:teuthology.orchestra.run.smithi191.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43724.log.gz 2024-09-14T06:10:18.469 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37590.log 2024-09-14T06:10:18.471 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.50015.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.50015.log.gz 2024-09-14T06:10:18.471 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37556.log 2024-09-14T06:10:18.471 INFO:teuthology.orchestra.run.smithi191.stderr: 2.3% -- replaced with /var/log/ceph/cluster2-client.mirror.37539.log.gz 2024-09-14T06:10:18.474 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.37590.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48858.log 2024-09-14T06:10:18.474 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.37556.log: 57.1% -- replaced with /var/log/ceph/cluster2-client.mirror.37590.log.gz 2024-09-14T06:10:18.474 INFO:teuthology.orchestra.run.smithi191.stderr: 2.3% -- replaced with /var/log/ceph/cluster2-client.mirror.37556.log.gz 2024-09-14T06:10:18.475 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61402.log 2024-09-14T06:10:18.475 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48858.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55752.log 2024-09-14T06:10:18.475 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.61402.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43306.log.gz 2024-09-14T06:10:18.475 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36747.log 2024-09-14T06:10:18.475 INFO:teuthology.orchestra.run.smithi191.stderr: 89.5% 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61402.log.gz 2024-09-14T06:10:18.475 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.48858.log.gz 2024-09-14T06:10:18.475 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46343.log 2024-09-14T06:10:18.475 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.55752.log: /var/log/ceph/cluster2-client.mirror.36747.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65687.log 2024-09-14T06:10:18.481 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46343.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.55752.log.gz 2024-09-14T06:10:18.481 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63662.log 2024-09-14T06:10:18.481 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65687.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68354.log 2024-09-14T06:10:18.481 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.63662.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65687.log.gz 2024-09-14T06:10:18.481 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65625.log 2024-09-14T06:10:18.481 INFO:teuthology.orchestra.run.smithi191.stderr: 73.6% -- replaced with /var/log/ceph/cluster2-client.mirror.36747.log.gz 2024-09-14T06:10:18.481 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.68354.log: 2.6% -- replaced with /var/log/ceph/cluster2-client.mirror.68354.log.gz 2024-09-14T06:10:18.481 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58395.log 2024-09-14T06:10:18.481 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65625.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63662.log.gz 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65625.log.gz 2024-09-14T06:10:18.481 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-14T06:10:18.481 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44899.log 2024-09-14T06:10:18.481 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.58395.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58395.log.gz 2024-09-14T06:10:18.482 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57469.log 2024-09-14T06:10:18.484 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.44899.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.44899.log.gz 2024-09-14T06:10:18.484 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47878.log 2024-09-14T06:10:18.484 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.57469.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57469.log.gz 2024-09-14T06:10:18.484 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45341.log 2024-09-14T06:10:18.487 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47878.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47878.log.gz 2024-09-14T06:10:18.487 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66783.log 2024-09-14T06:10:18.487 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45341.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45341.log.gz 2024-09-14T06:10:18.487 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40508.log 2024-09-14T06:10:18.487 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.66783.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66783.log.gz 2024-09-14T06:10:18.489 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47399.log 2024-09-14T06:10:18.490 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.40508.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.40508.log.gz 2024-09-14T06:10:18.490 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44025.log 2024-09-14T06:10:18.491 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47399.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47399.log.gz 2024-09-14T06:10:18.491 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69561.log 2024-09-14T06:10:18.491 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.44025.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43486.log 2024-09-14T06:10:18.493 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.69561.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.69561.log.gz 2024-09-14T06:10:18.493 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47118.log 2024-09-14T06:10:18.496 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.43486.log: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.43486.log.gz 2024-09-14T06:10:18.496 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39649.log 2024-09-14T06:10:18.496 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47118.log: 80.1% -- replaced with /var/log/ceph/cluster2-client.mirror.47118.log.gz 2024-09-14T06:10:18.496 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55410.log 2024-09-14T06:10:18.496 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.39649.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66738.log 2024-09-14T06:10:18.500 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.55410.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.55410.log.gz 2024-09-14T06:10:18.500 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50216.log 2024-09-14T06:10:18.500 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.66738.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66738.log.gz 2024-09-14T06:10:18.500 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58315.log 2024-09-14T06:10:18.501 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.50216.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.50216.log.gz 2024-09-14T06:10:18.501 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46284.log 2024-09-14T06:10:18.501 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.58315.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44025.log.gz 2024-09-14T06:10:18.502 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55256.log 2024-09-14T06:10:18.502 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46284.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46284.log.gz 2024-09-14T06:10:18.505 INFO:teuthology.orchestra.run.smithi191.stderr: 87.7%gzip/var/log/ceph/cluster2-client.mirror.55256.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47419.log 2024-09-14T06:10:18.505 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.39649.log.gz 2024-09-14T06:10:18.505 INFO:teuthology.orchestra.run.smithi191.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58315.log.gz 2024-09-14T06:10:18.505 INFO:teuthology.orchestra.run.smithi191.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.55256.log.gz 2024-09-14T06:10:18.505 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63972.log 2024-09-14T06:10:18.505 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.47419.log: -- /var/log/ceph/cluster2-client.mirror.71156.log 2024-09-14T06:10:18.506 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47419.log.gz 2024-09-14T06:10:18.506 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.63972.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62049.log 2024-09-14T06:10:18.508 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.71156.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.71156.log.gz 2024-09-14T06:10:18.508 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63621.log 2024-09-14T06:10:18.508 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.62049.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63972.log.gz 2024-09-14T06:10:18.508 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40251.log 2024-09-14T06:10:18.510 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.63621.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62049.log.gz 2024-09-14T06:10:18.511 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71176.log 2024-09-14T06:10:18.511 INFO:teuthology.orchestra.run.smithi191.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63621.log.gz 2024-09-14T06:10:18.511 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63015.log 2024-09-14T06:10:18.511 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.40251.log: /var/log/ceph/cluster2-client.mirror.71176.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63289.log 2024-09-14T06:10:18.511 INFO:teuthology.orchestra.run.smithi191.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.71176.log.gz 2024-09-14T06:10:18.511 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.63015.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70398.log 2024-09-14T06:10:18.513 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.63289.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63289.log.gz 2024-09-14T06:10:18.516 INFO:teuthology.orchestra.run.smithi191.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63015.log.gz 2024-09-14T06:10:18.516 INFO:teuthology.orchestra.run.smithi191.stderr: 87.8%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53480.log 2024-09-14T06:10:18.516 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.40251.log.gz 2024-09-14T06:10:18.516 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72508.log 2024-09-14T06:10:18.519 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.70398.log: /var/log/ceph/cluster1-client.mirror.53480.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.70398.log.gz 2024-09-14T06:10:18.519 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63310.log 2024-09-14T06:10:18.519 INFO:teuthology.orchestra.run.smithi191.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53480.log.gz 2024-09-14T06:10:18.520 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.72508.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72508.log.gz 2024-09-14T06:10:18.520 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32580.log 2024-09-14T06:10:18.520 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.63310.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65893.log 2024-09-14T06:10:18.520 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.32580.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32580.log.gz 2024-09-14T06:10:18.520 INFO:teuthology.orchestra.run.smithi191.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63310.log.gz 2024-09-14T06:10:18.520 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62233.log 2024-09-14T06:10:18.520 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56423.log 2024-09-14T06:10:18.523 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65893.log: /var/log/ceph/cluster1-client.mirror.62233.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.62233.log.gz 2024-09-14T06:10:18.523 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61289.log 2024-09-14T06:10:18.523 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.56423.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56423.log.gz 2024-09-14T06:10:18.523 INFO:teuthology.orchestra.run.smithi191.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65893.log.gz 2024-09-14T06:10:18.523 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54316.log 2024-09-14T06:10:18.526 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.61289.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61289.log.gz 2024-09-14T06:10:18.527 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64608.log 2024-09-14T06:10:18.527 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39506.log 2024-09-14T06:10:18.527 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64608.log: /var/log/ceph/cluster1-client.mirror.54316.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54316.log.gz 2024-09-14T06:10:18.527 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42816.log 2024-09-14T06:10:18.527 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.39506.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64034.log 2024-09-14T06:10:18.528 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.42816.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64608.log.gz 2024-09-14T06:10:18.529 INFO:teuthology.orchestra.run.smithi191.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.39506.log.gz 2024-09-14T06:10:18.529 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44220.log 2024-09-14T06:10:18.533 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64034.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64034.log.gz 2024-09-14T06:10:18.533 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66265.log 2024-09-14T06:10:18.533 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.44220.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42816.log.gz 2024-09-14T06:10:18.535 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64282.log 2024-09-14T06:10:18.535 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.66265.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66265.log.gz 2024-09-14T06:10:18.535 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42619.log 2024-09-14T06:10:18.552 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64282.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64282.log.gz 2024-09-14T06:10:18.553 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65535.log 2024-09-14T06:10:18.553 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.42619.log: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.42619.log.gz 2024-09-14T06:10:18.553 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58178.log 2024-09-14T06:10:18.553 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65535.log: 88.4% 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44220.log.gz 2024-09-14T06:10:18.553 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.65535.log.gz 2024-09-14T06:10:18.553 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.65535.log 2024-09-14T06:10:18.553 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.58178.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65749.log 2024-09-14T06:10:18.553 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58178.log.gz 2024-09-14T06:10:18.553 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.0.65535.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52716.log 2024-09-14T06:10:18.553 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65749.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65749.log.gz 2024-09-14T06:10:18.553 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70340.log 2024-09-14T06:10:18.553 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.52716.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52716.log.gz 2024-09-14T06:10:18.554 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49967.log 2024-09-14T06:10:18.554 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.70340.log: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.0.65535.log.gz 2024-09-14T06:10:18.554 INFO:teuthology.orchestra.run.smithi191.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.70340.log.gz 2024-09-14T06:10:18.554 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43876.log 2024-09-14T06:10:18.554 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38680.log 2024-09-14T06:10:18.554 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.43876.log: /var/log/ceph/cluster1-client.mirror.49967.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.49967.log.gz 2024-09-14T06:10:18.554 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69647.log 2024-09-14T06:10:18.554 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.38680.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45616.log 2024-09-14T06:10:18.554 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.69647.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69647.log.gz 2024-09-14T06:10:18.554 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41086.log 2024-09-14T06:10:18.554 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45616.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.45616.log.gz 2024-09-14T06:10:18.554 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48762.log 2024-09-14T06:10:18.554 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.41086.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41086.log.gz 2024-09-14T06:10:18.555 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47197.log 2024-09-14T06:10:18.555 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48762.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48762.log.gz 2024-09-14T06:10:18.555 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.39304.log 2024-09-14T06:10:18.555 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47197.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43876.log.gz 2024-09-14T06:10:18.555 INFO:teuthology.orchestra.run.smithi191.stderr: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47197.log.gz 2024-09-14T06:10:18.555 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60665.log 2024-09-14T06:10:18.555 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.0.39304.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44798.log 2024-09-14T06:10:18.555 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.60665.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.60665.log.gz 2024-09-14T06:10:18.556 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62747.log 2024-09-14T06:10:18.557 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.44798.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44798.log.gz 2024-09-14T06:10:18.560 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44678.log 2024-09-14T06:10:18.560 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52017.log 2024-09-14T06:10:18.561 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.62747.log: /var/log/ceph/cluster2-client.mirror.44678.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62747.log.gz 2024-09-14T06:10:18.562 INFO:teuthology.orchestra.run.smithi191.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.40272.log.gz 2024-09-14T06:10:18.566 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50537.log 2024-09-14T06:10:18.566 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.52017.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44678.log.gz 2024-09-14T06:10:18.566 INFO:teuthology.orchestra.run.smithi191.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52017.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63179.log 2024-09-14T06:10:18.566 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-14T06:10:18.566 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43595.log 2024-09-14T06:10:18.566 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.63179.log: /var/log/ceph/cluster1-client.mirror.50537.log: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.63179.log.gz 2024-09-14T06:10:18.566 INFO:teuthology.orchestra.run.smithi191.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.50537.log.gz 2024-09-14T06:10:18.566 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57228.log 2024-09-14T06:10:18.566 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53889.log 2024-09-14T06:10:18.566 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.57228.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57228.log.gz 2024-09-14T06:10:18.575 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.43595.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43595.log.gz 2024-09-14T06:10:18.575 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72236.log 2024-09-14T06:10:18.575 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.53889.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53889.log.gz 2024-09-14T06:10:18.579 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49197.log 2024-09-14T06:10:18.580 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.72236.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.72236.log.gz 2024-09-14T06:10:18.581 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61679.log 2024-09-14T06:10:18.585 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49197.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49197.log.gz 2024-09-14T06:10:18.585 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72083.log 2024-09-14T06:10:18.585 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.61679.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49700.log 2024-09-14T06:10:18.585 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61679.log.gz 2024-09-14T06:10:18.585 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.72083.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36659.log 2024-09-14T06:10:18.585 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.49700.log: 72.9% -- replaced with /var/log/ceph/cluster1-client.mirror.49700.log.gz 2024-09-14T06:10:18.585 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40179.log 2024-09-14T06:10:18.585 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.36659.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36659.log.gz 2024-09-14T06:10:18.585 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67323.log 2024-09-14T06:10:18.587 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.40179.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48396.log 2024-09-14T06:10:18.588 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.67323.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.67323.log.gz 2024-09-14T06:10:18.588 INFO:teuthology.orchestra.run.smithi191.stderr: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.72083.log.gz 2024-09-14T06:10:18.592 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48396.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71434.log 2024-09-14T06:10:18.592 INFO:teuthology.orchestra.run.smithi191.stderr: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.40179.log.gz 2024-09-14T06:10:18.592 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45815.log 2024-09-14T06:10:18.593 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.71434.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.71434.log.gz 2024-09-14T06:10:18.593 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39873.log 2024-09-14T06:10:18.594 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45815.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45815.log.gz 2024-09-14T06:10:18.594 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62394.log 2024-09-14T06:10:18.596 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.39873.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47089.log 2024-09-14T06:10:18.596 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.62394.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62394.log.gz 2024-09-14T06:10:18.597 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52491.log 2024-09-14T06:10:18.614 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47089.log: /var/log/ceph/cluster1-client.mirror.52491.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.52491.log.gz 2024-09-14T06:10:18.617 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40732.log 2024-09-14T06:10:18.622 INFO:teuthology.orchestra.run.smithi191.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45675.log.gz 2024-09-14T06:10:18.623 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50582.log 2024-09-14T06:10:18.628 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.40732.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.40732.log.gz 2024-09-14T06:10:18.630 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68389.log 2024-09-14T06:10:18.633 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.50582.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.50582.log.gz 2024-09-14T06:10:18.634 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52110.log 2024-09-14T06:10:18.635 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.68389.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.68389.log.gz 2024-09-14T06:10:18.636 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65209.log 2024-09-14T06:10:18.637 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.52110.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52110.log.gz 2024-09-14T06:10:18.638 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72845.log 2024-09-14T06:10:18.638 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65209.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65209.log.gz 2024-09-14T06:10:18.639 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54043.log 2024-09-14T06:10:18.639 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.72845.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72845.log.gz 2024-09-14T06:10:18.640 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71956.log 2024-09-14T06:10:18.640 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.54043.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.54043.log.gz 2024-09-14T06:10:18.641 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45292.log 2024-09-14T06:10:18.641 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.71956.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.71956.log.gz 2024-09-14T06:10:18.642 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60134.log 2024-09-14T06:10:18.663 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45292.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42732.log 2024-09-14T06:10:18.664 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.60134.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.60134.log.gz 2024-09-14T06:10:18.676 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60966.log 2024-09-14T06:10:18.677 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.42732.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.42732.log.gz 2024-09-14T06:10:18.679 INFO:teuthology.orchestra.run.smithi191.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.38680.log.gz 2024-09-14T06:10:18.689 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37419.log 2024-09-14T06:10:18.690 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.60966.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.60966.log.gz 2024-09-14T06:10:18.692 INFO:teuthology.orchestra.run.smithi191.stderr: 93.7% -- replaced with /var/log/ceph/cluster1-client.mirror.0.39304.log.gz 2024-09-14T06:10:18.697 INFO:teuthology.orchestra.run.smithi191.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.70813.log.gz 2024-09-14T06:10:18.699 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46929.log 2024-09-14T06:10:18.702 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.37419.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37419.log.gz 2024-09-14T06:10:18.703 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43414.log 2024-09-14T06:10:18.706 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46929.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49138.log.gz 2024-09-14T06:10:18.722 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32679.log 2024-09-14T06:10:18.724 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.43414.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57080.log 2024-09-14T06:10:18.724 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.32679.log: 89.5% 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32679.log.gz 2024-09-14T06:10:18.724 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.43414.log.gz 2024-09-14T06:10:18.724 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66828.log 2024-09-14T06:10:18.725 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.57080.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57080.log.gz 2024-09-14T06:10:18.732 INFO:teuthology.orchestra.run.smithi191.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46343.log.gz 2024-09-14T06:10:18.740 INFO:teuthology.orchestra.run.smithi191.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49498.log.gz 2024-09-14T06:10:18.745 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.40951.log 2024-09-14T06:10:18.745 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.66828.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71198.log 2024-09-14T06:10:18.745 INFO:teuthology.orchestra.run.smithi191.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66828.log.gz 2024-09-14T06:10:18.745 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.0.40951.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52965.log 2024-09-14T06:10:18.745 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.71198.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55714.log 2024-09-14T06:10:18.750 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.52965.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52965.log.gz 2024-09-14T06:10:18.750 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46163.log 2024-09-14T06:10:18.752 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.55714.log: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.0.40951.log.gz 2024-09-14T06:10:18.752 INFO:teuthology.orchestra.run.smithi191.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.55714.log.gz 2024-09-14T06:10:18.752 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72971.log 2024-09-14T06:10:18.752 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60713.log 2024-09-14T06:10:18.753 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46163.log: /var/log/ceph/cluster2-client.admin.72971.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72971.log.gz 2024-09-14T06:10:18.753 INFO:teuthology.orchestra.run.smithi191.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46163.log.gz 2024-09-14T06:10:18.761 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60460.log 2024-09-14T06:10:18.761 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.60713.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66220.log 2024-09-14T06:10:18.761 INFO:teuthology.orchestra.run.smithi191.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60713.log.gz 2024-09-14T06:10:18.761 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.60460.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69887.log 2024-09-14T06:10:18.761 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.66220.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.66220.log.gz 2024-09-14T06:10:18.761 INFO:teuthology.orchestra.run.smithi191.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60460.log.gz 2024-09-14T06:10:18.761 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47542.log 2024-09-14T06:10:18.761 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.69887.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51414.log 2024-09-14T06:10:18.761 INFO:teuthology.orchestra.run.smithi191.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69887.log.gz 2024-09-14T06:10:18.761 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47542.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47542.log.gz 2024-09-14T06:10:18.761 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63993.log 2024-09-14T06:10:18.762 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.51414.log: -- /var/log/ceph/cluster1-client.mirror.51816.log 2024-09-14T06:10:18.762 INFO:teuthology.orchestra.run.smithi191.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.51414.log.gz 2024-09-14T06:10:18.762 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.63993.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48937.log 2024-09-14T06:10:18.762 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.51816.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.63993.log.gz 87.4% 2024-09-14T06:10:18.762 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.51816.log.gz 2024-09-14T06:10:18.762 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56664.log 2024-09-14T06:10:18.762 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48937.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50104.log 2024-09-14T06:10:18.762 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.56664.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56664.log.gz 2024-09-14T06:10:18.762 INFO:teuthology.orchestra.run.smithi191.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48937.log.gz 2024-09-14T06:10:18.762 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42320.log 2024-09-14T06:10:18.764 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.50104.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.50104.log.gz 2024-09-14T06:10:18.764 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67525.log 2024-09-14T06:10:18.764 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.42320.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70298.log 2024-09-14T06:10:18.766 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.67525.log: 85.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67525.log.gz 2024-09-14T06:10:18.767 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39851.log 2024-09-14T06:10:18.767 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.70298.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70298.log.gz 2024-09-14T06:10:18.768 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43438.log 2024-09-14T06:10:18.769 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.39851.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69726.log 2024-09-14T06:10:18.770 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.43438.log: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.43438.log.gz 2024-09-14T06:10:18.772 INFO:teuthology.orchestra.run.smithi191.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.39851.log.gz 2024-09-14T06:10:18.773 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47898.log 2024-09-14T06:10:18.773 INFO:teuthology.orchestra.run.smithi191.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42320.log.gz 2024-09-14T06:10:18.773 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.69726.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35651.log 2024-09-14T06:10:18.775 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47898.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.69726.log.gz 2024-09-14T06:10:18.775 INFO:teuthology.orchestra.run.smithi191.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45758.log.gz 2024-09-14T06:10:18.775 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54646.log 2024-09-14T06:10:18.775 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.35651.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35651.log.gz 2024-09-14T06:10:18.775 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45048.log 2024-09-14T06:10:18.775 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.54646.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63828.log 2024-09-14T06:10:18.776 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45048.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47898.log.gz 2024-09-14T06:10:18.776 INFO:teuthology.orchestra.run.smithi191.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.54646.log.gz 2024-09-14T06:10:18.777 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64504.log 2024-09-14T06:10:18.777 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.63828.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63828.log.gz 2024-09-14T06:10:18.778 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51076.log 2024-09-14T06:10:18.781 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64504.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64504.log.gz 2024-09-14T06:10:18.781 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.70185.log 2024-09-14T06:10:18.781 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.51076.log: 74.0% -- replaced with /var/log/ceph/cluster2-client.mirror.51076.log.gz 2024-09-14T06:10:18.782 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43194.log 2024-09-14T06:10:18.782 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59316.log 2024-09-14T06:10:18.782 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.43194.log: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.43194.log.gz 2024-09-14T06:10:18.785 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.1.70185.log: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.1.70185.log.gz 2024-09-14T06:10:18.786 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72063.log 2024-09-14T06:10:18.787 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.59316.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.59316.log.gz 2024-09-14T06:10:18.789 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49160.log 2024-09-14T06:10:18.790 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.72063.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71599.log 2024-09-14T06:10:18.790 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49160.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49160.log.gz 2024-09-14T06:10:18.794 INFO:teuthology.orchestra.run.smithi191.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.72063.log.gz 2024-09-14T06:10:18.794 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66850.log 2024-09-14T06:10:18.794 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.71599.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.71599.log.gz 2024-09-14T06:10:18.794 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48079.log 2024-09-14T06:10:18.794 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.66850.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66850.log.gz 2024-09-14T06:10:18.794 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50810.log 2024-09-14T06:10:18.794 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48079.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64857.log 2024-09-14T06:10:18.795 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.50810.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.50810.log.gz 2024-09-14T06:10:18.803 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40776.log 2024-09-14T06:10:18.811 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64857.log: 90.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40712.log 2024-09-14T06:10:18.811 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.64857.log.gz 2024-09-14T06:10:18.820 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.40776.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.70875.log 2024-09-14T06:10:18.829 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.40712.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27668.log 2024-09-14T06:10:18.835 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.0.70875.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.40712.log.gz 2024-09-14T06:10:18.837 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69603.log 2024-09-14T06:10:18.838 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph.tmp-client.admin.27668.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48396.log.gz 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27668.log.gz 2024-09-14T06:10:18.838 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-14T06:10:18.838 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66085.log 2024-09-14T06:10:18.843 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.69603.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69603.log.gz 2024-09-14T06:10:18.843 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67908.log 2024-09-14T06:10:18.843 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.66085.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.66085.log.gz 2024-09-14T06:10:18.843 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45400.log 2024-09-14T06:10:18.843 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.67908.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67908.log.gz 2024-09-14T06:10:18.843 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50939.log 2024-09-14T06:10:18.845 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45400.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.0.70875.log.gz 2024-09-14T06:10:18.855 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54099.log 2024-09-14T06:10:18.856 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.50939.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50939.log.gz 2024-09-14T06:10:18.868 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72677.log 2024-09-14T06:10:18.869 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.54099.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.54099.log.gz 2024-09-14T06:10:18.893 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59143.log 2024-09-14T06:10:18.893 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.72677.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72677.log.gz 2024-09-14T06:10:18.906 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60159.log 2024-09-14T06:10:18.906 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.59143.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.59143.log.gz 2024-09-14T06:10:18.920 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65126.log 2024-09-14T06:10:18.920 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.60159.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60159.log.gz 2024-09-14T06:10:18.929 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49456.log 2024-09-14T06:10:18.934 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65126.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65126.log.gz 2024-09-14T06:10:18.943 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45716.log 2024-09-14T06:10:18.944 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49456.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49456.log.gz 2024-09-14T06:10:18.952 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70754.log 2024-09-14T06:10:18.966 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45716.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45716.log.gz 2024-09-14T06:10:18.968 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57122.log 2024-09-14T06:10:18.969 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.70754.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.70754.log.gz 2024-09-14T06:10:18.977 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64324.log 2024-09-14T06:10:18.986 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.57122.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57122.log.gz 2024-09-14T06:10:18.997 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55555.log 2024-09-14T06:10:18.997 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64324.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64324.log.gz 2024-09-14T06:10:19.005 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56302.log 2024-09-14T06:10:19.015 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.55555.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55555.log.gz 2024-09-14T06:10:19.021 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49833.log 2024-09-14T06:10:19.022 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.56302.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56302.log.gz 2024-09-14T06:10:19.030 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35693.log 2024-09-14T06:10:19.039 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.49833.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51885.log 2024-09-14T06:10:19.040 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.35693.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35693.log.gz 2024-09-14T06:10:19.043 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.49833.log.gz 2024-09-14T06:10:19.048 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35306.log 2024-09-14T06:10:19.062 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.51885.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51885.log.gz 2024-09-14T06:10:19.068 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49791.log 2024-09-14T06:10:19.068 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.35306.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35306.log.gz 2024-09-14T06:10:19.077 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39125.log 2024-09-14T06:10:19.078 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49791.log: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.49791.log.gz 2024-09-14T06:10:19.085 INFO:teuthology.orchestra.run.smithi191.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46929.log.gz 2024-09-14T06:10:19.085 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36281.log 2024-09-14T06:10:19.087 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.39125.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.69155.log 2024-09-14T06:10:19.087 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.36281.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36281.log.gz 2024-09-14T06:10:19.087 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50353.log 2024-09-14T06:10:19.091 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.0.69155.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.39125.log.gz 2024-09-14T06:10:19.092 INFO:teuthology.orchestra.run.smithi191.stderr: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.0.69155.log.gz 2024-09-14T06:10:19.092 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51435.log 2024-09-14T06:10:19.097 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.50353.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.50353.log.gz 2024-09-14T06:10:19.098 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68807.log 2024-09-14T06:10:19.107 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.51435.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.51435.log.gz 2024-09-14T06:10:19.112 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53118.log 2024-09-14T06:10:19.112 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.68807.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68807.log.gz 2024-09-14T06:10:19.116 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42924.log 2024-09-14T06:10:19.116 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.53118.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53118.log.gz 2024-09-14T06:10:19.117 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41603.log 2024-09-14T06:10:19.121 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.42924.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.42924.log.gz 2024-09-14T06:10:19.121 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68020.log 2024-09-14T06:10:19.126 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.41603.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.41603.log.gz 2024-09-14T06:10:19.126 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45894.log 2024-09-14T06:10:19.134 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.68020.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.68020.log.gz 2024-09-14T06:10:19.135 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46781.log 2024-09-14T06:10:19.135 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45894.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45894.log.gz 2024-09-14T06:10:19.136 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45653.log 2024-09-14T06:10:19.153 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46781.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37624.log 2024-09-14T06:10:19.153 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45653.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45653.log.gz 2024-09-14T06:10:19.161 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62161.log 2024-09-14T06:10:19.176 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.37624.log: 54.2% -- replaced with /var/log/ceph/cluster2-client.mirror.37624.log.gz 2024-09-14T06:10:19.182 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.54815.log 2024-09-14T06:10:19.183 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.62161.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62161.log.gz 2024-09-14T06:10:19.193 INFO:teuthology.orchestra.run.smithi191.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45292.log.gz 2024-09-14T06:10:19.194 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43242.log 2024-09-14T06:10:19.196 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.0.54815.log: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.0.54815.log.gz 2024-09-14T06:10:19.196 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57662.log 2024-09-14T06:10:19.198 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.43242.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.43242.log.gz 2024-09-14T06:10:19.199 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59769.log 2024-09-14T06:10:19.199 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.57662.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57662.log.gz 2024-09-14T06:10:19.199 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62809.log 2024-09-14T06:10:19.201 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.59769.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59769.log.gz 2024-09-14T06:10:19.201 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59236.log 2024-09-14T06:10:19.203 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.62809.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.62809.log.gz 2024-09-14T06:10:19.203 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67075.log 2024-09-14T06:10:19.205 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.59236.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59236.log.gz 2024-09-14T06:10:19.205 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69993.log 2024-09-14T06:10:19.206 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.67075.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67075.log.gz 2024-09-14T06:10:19.206 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57100.log 2024-09-14T06:10:19.207 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.69993.log: 77.1% -- replaced with /var/log/ceph/cluster2-client.mirror.69993.log.gz 2024-09-14T06:10:19.208 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51927.log 2024-09-14T06:10:19.210 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.57100.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.57100.log.gz 2024-09-14T06:10:19.210 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70791.log 2024-09-14T06:10:19.212 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.51927.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51927.log.gz 2024-09-14T06:10:19.212 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45358.log 2024-09-14T06:10:19.212 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.70791.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.70791.log.gz 2024-09-14T06:10:19.213 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64692.log 2024-09-14T06:10:19.215 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45358.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45358.log.gz 2024-09-14T06:10:19.215 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68763.log 2024-09-14T06:10:19.217 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64692.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64692.log.gz 2024-09-14T06:10:19.217 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46951.log 2024-09-14T06:10:19.219 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.68763.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68763.log.gz 2024-09-14T06:10:19.219 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61543.log 2024-09-14T06:10:19.219 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46951.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46951.log.gz 2024-09-14T06:10:19.220 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61888.log 2024-09-14T06:10:19.222 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.61543.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61543.log.gz 2024-09-14T06:10:19.223 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39959.log 2024-09-14T06:10:19.224 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.61888.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.61888.log.gz 2024-09-14T06:10:19.224 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47010.log 2024-09-14T06:10:19.226 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.39959.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46744.log 2024-09-14T06:10:19.230 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47010.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.71198.log.gz 2024-09-14T06:10:19.233 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54668.log 2024-09-14T06:10:19.272 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46744.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41530.log 2024-09-14T06:10:19.276 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.54668.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.54668.log.gz 2024-09-14T06:10:19.276 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61426.log 2024-09-14T06:10:19.281 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.41530.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41530.log.gz 2024-09-14T06:10:19.282 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64158.log 2024-09-14T06:10:19.286 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.61426.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.61426.log.gz 2024-09-14T06:10:19.286 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41874.log 2024-09-14T06:10:19.295 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64158.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52423.log 2024-09-14T06:10:19.296 INFO:teuthology.orchestra.run.smithi191.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64158.log.gz 2024-09-14T06:10:19.296 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.41874.log: 81.8% -- replaced with /var/log/ceph/cluster1-client.mirror.41874.log.gz 2024-09-14T06:10:19.299 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67773.log 2024-09-14T06:10:19.299 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.52423.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52423.log.gz 2024-09-14T06:10:19.301 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41134.log 2024-09-14T06:10:19.304 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.67773.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67773.log.gz 2024-09-14T06:10:19.304 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63704.log 2024-09-14T06:10:19.305 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.41134.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.41134.log.gz 2024-09-14T06:10:19.308 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56148.log 2024-09-14T06:10:19.314 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.63704.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63704.log.gz 2024-09-14T06:10:19.314 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64630.log 2024-09-14T06:10:19.314 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.56148.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.56148.log.gz 2024-09-14T06:10:19.314 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51165.log 2024-09-14T06:10:19.322 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64630.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64630.log.gz 2024-09-14T06:10:19.323 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55015.log 2024-09-14T06:10:19.324 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.51165.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.51165.log.gz 2024-09-14T06:10:19.324 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46722.log 2024-09-14T06:10:19.324 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.55015.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.55015.log.gz 2024-09-14T06:10:19.328 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.36849.log 2024-09-14T06:10:19.331 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46722.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46722.log.gz 2024-09-14T06:10:19.333 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42753.log 2024-09-14T06:10:19.333 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.3.36849.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51342.log 2024-09-14T06:10:19.358 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.42753.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42753.log.gz 2024-09-14T06:10:19.360 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53616.log 2024-09-14T06:10:19.361 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.51342.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.51342.log.gz 2024-09-14T06:10:19.361 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47559.log 2024-09-14T06:10:19.369 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.53616.log: 93.7% -- replaced with /var/log/ceph/cluster1-client.mirror.3.36849.log.gz 2024-09-14T06:10:19.369 INFO:teuthology.orchestra.run.smithi191.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53616.log.gz 2024-09-14T06:10:19.369 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61382.log 2024-09-14T06:10:19.369 INFO:teuthology.orchestra.run.smithi191.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46781.log.gz 2024-09-14T06:10:19.371 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47559.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47559.log.gz 2024-09-14T06:10:19.371 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62281.log 2024-09-14T06:10:19.371 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.61382.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61382.log.gz 2024-09-14T06:10:19.372 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62004.log 2024-09-14T06:10:19.375 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.62281.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.62281.log.gz 2024-09-14T06:10:19.375 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60484.log 2024-09-14T06:10:19.376 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.62004.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62004.log.gz 2024-09-14T06:10:19.376 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63910.log 2024-09-14T06:10:19.382 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.60484.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60484.log.gz 2024-09-14T06:10:19.382 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44005.log 2024-09-14T06:10:19.390 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.63910.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63910.log.gz 2024-09-14T06:10:19.391 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51206.log 2024-09-14T06:10:19.397 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.44005.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45400.log.gz 2024-09-14T06:10:19.398 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72887.log 2024-09-14T06:10:19.401 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.51206.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.51206.log.gz 2024-09-14T06:10:19.401 INFO:teuthology.orchestra.run.smithi191.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44005.log.gz 2024-09-14T06:10:19.402 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55535.log 2024-09-14T06:10:19.403 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.72887.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72887.log.gz 2024-09-14T06:10:19.403 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54993.log 2024-09-14T06:10:19.404 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.55535.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.55535.log.gz 2024-09-14T06:10:19.408 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62462.log 2024-09-14T06:10:19.410 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47155.log 2024-09-14T06:10:19.410 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.62462.log: /var/log/ceph/cluster1-client.mirror.54993.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.62462.log.gz 2024-09-14T06:10:19.410 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39745.log 2024-09-14T06:10:19.411 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47155.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47155.log.gz 2024-09-14T06:10:19.413 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59488.log 2024-09-14T06:10:19.413 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.39745.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35533.log 2024-09-14T06:10:19.415 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.59488.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59488.log.gz 2024-09-14T06:10:19.415 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41644.log 2024-09-14T06:10:19.415 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.35533.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35533.log.gz 2024-09-14T06:10:19.416 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37488.log 2024-09-14T06:10:19.416 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.41644.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.39745.log.gz 2024-09-14T06:10:19.417 INFO:teuthology.orchestra.run.smithi191.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47089.log.gz 2024-09-14T06:10:19.419 INFO:teuthology.orchestra.run.smithi191.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.54993.log.gz 2024-09-14T06:10:19.419 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72286.log 2024-09-14T06:10:19.420 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.37488.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48682.log 2024-09-14T06:10:19.424 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.72286.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.72286.log.gz 2024-09-14T06:10:19.424 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66288.log 2024-09-14T06:10:19.424 INFO:teuthology.orchestra.run.smithi191.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.41644.log.gz 2024-09-14T06:10:19.424 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48682.log: 80.1% -- replaced with /var/log/ceph/cluster2-client.admin.37488.log.gz 2024-09-14T06:10:19.424 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56254.log 2024-09-14T06:10:19.425 INFO:teuthology.orchestra.run.smithi191.stderr: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48682.log.gz 2024-09-14T06:10:19.425 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.66288.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66288.log.gz 2024-09-14T06:10:19.425 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47680.log 2024-09-14T06:10:19.425 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69351.log 2024-09-14T06:10:19.425 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47680.log: /var/log/ceph/cluster1-client.mirror.56254.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.56254.log.gz 2024-09-14T06:10:19.426 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69972.log 2024-09-14T06:10:19.427 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.69351.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69351.log.gz 2024-09-14T06:10:19.427 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69910.log 2024-09-14T06:10:19.429 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.69972.log: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.69972.log.gz 2024-09-14T06:10:19.429 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48295.log 2024-09-14T06:10:19.430 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53235.log 2024-09-14T06:10:19.431 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48295.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48295.log.gz 2024-09-14T06:10:19.431 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59789.log 2024-09-14T06:10:19.434 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.53235.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53235.log.gz 2024-09-14T06:10:19.434 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.69910.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46301.log 2024-09-14T06:10:19.434 INFO:teuthology.orchestra.run.smithi191.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69910.log.gz 2024-09-14T06:10:19.434 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.59789.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63052.log 2024-09-14T06:10:19.436 INFO:teuthology.orchestra.run.smithi191.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59789.log.gz 2024-09-14T06:10:19.436 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47861.log 2024-09-14T06:10:19.438 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.63052.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.63052.log.gz 2024-09-14T06:10:19.438 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46301.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71791.log 2024-09-14T06:10:19.438 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47861.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47861.log.gz 2024-09-14T06:10:19.439 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40576.log 2024-09-14T06:10:19.442 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.71791.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.71791.log.gz 2024-09-14T06:10:19.442 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63372.log 2024-09-14T06:10:19.442 INFO:teuthology.orchestra.run.smithi191.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.40776.log.gz 2024-09-14T06:10:19.442 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.40576.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62974.log 2024-09-14T06:10:19.442 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.63372.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.40576.log.gz 2024-09-14T06:10:19.442 INFO:teuthology.orchestra.run.smithi191.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46301.log.gz 2024-09-14T06:10:19.442 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71620.log 2024-09-14T06:10:19.442 INFO:teuthology.orchestra.run.smithi191.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63372.log.gz 2024-09-14T06:10:19.444 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.62974.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71555.log 2024-09-14T06:10:19.444 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.71620.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62974.log.gz 2024-09-14T06:10:19.444 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.log 2024-09-14T06:10:19.444 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.71555.log: 91.3% -- replaced with /var/log/ceph/cluster2-client.mirror.71620.log.gz 2024-09-14T06:10:19.445 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70440.log 2024-09-14T06:10:19.450 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2.log: 87.0% -- replaced with /var/log/ceph/cluster2.log.gz 2024-09-14T06:10:19.450 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57180.log 2024-09-14T06:10:19.452 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.70440.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56085.log 2024-09-14T06:10:19.453 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.57180.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.57180.log.gz 2024-09-14T06:10:19.453 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.36929.log 2024-09-14T06:10:19.457 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.56085.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.56085.log.gz 2024-09-14T06:10:19.459 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60296.log 2024-09-14T06:10:19.459 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.3.36929.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42578.log 2024-09-14T06:10:19.463 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.60296.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.60296.log.gz 2024-09-14T06:10:19.464 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64365.log 2024-09-14T06:10:19.469 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.42578.log: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42578.log.gz 2024-09-14T06:10:19.471 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43348.log 2024-09-14T06:10:19.472 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64365.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64365.log.gz 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48079.log.gz 2024-09-14T06:10:19.472 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-14T06:10:19.473 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53592.log 2024-09-14T06:10:19.473 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.43348.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.43348.log.gz 2024-09-14T06:10:19.475 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40914.log 2024-09-14T06:10:19.475 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.53592.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53592.log.gz 2024-09-14T06:10:19.475 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72111.log 2024-09-14T06:10:19.476 INFO:teuthology.orchestra.run.smithi191.stderr: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.3.36929.log.gz 2024-09-14T06:10:19.478 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.40914.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45956.log 2024-09-14T06:10:19.478 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.72111.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72111.log.gz 2024-09-14T06:10:19.478 INFO:teuthology.orchestra.run.smithi191.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.40914.log.gz 2024-09-14T06:10:19.478 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53688.log 2024-09-14T06:10:19.479 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36239.log 2024-09-14T06:10:19.480 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45956.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45956.log.gz 2024-09-14T06:10:19.480 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68178.log 2024-09-14T06:10:19.481 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.36239.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36239.log.gz 2024-09-14T06:10:19.481 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51455.log 2024-09-14T06:10:19.483 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.68178.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68178.log.gz 2024-09-14T06:10:19.483 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42342.log 2024-09-14T06:10:19.486 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.51455.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.51455.log.gz 2024-09-14T06:10:19.486 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.53688.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47256.log 2024-09-14T06:10:19.486 INFO:teuthology.orchestra.run.smithi191.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53688.log.gz 2024-09-14T06:10:19.486 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.42342.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55386.log 2024-09-14T06:10:19.487 INFO:teuthology.orchestra.run.smithi191.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.42342.log.gz 2024-09-14T06:10:19.487 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67435.log 2024-09-14T06:10:19.488 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.55386.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55386.log.gz 2024-09-14T06:10:19.488 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56365.log 2024-09-14T06:10:19.490 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.67435.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67435.log.gz 2024-09-14T06:10:19.490 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50192.log 2024-09-14T06:10:19.491 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.56365.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.56365.log.gz 2024-09-14T06:10:19.491 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58877.log 2024-09-14T06:10:19.493 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.50192.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50192.log.gz 2024-09-14T06:10:19.493 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58795.log 2024-09-14T06:10:19.494 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.58877.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58877.log.gz 2024-09-14T06:10:19.494 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63538.log 2024-09-14T06:10:19.496 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.58795.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.58795.log.gz 2024-09-14T06:10:19.496 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48514.log 2024-09-14T06:10:19.497 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.63538.log: /var/log/ceph/cluster2-client.mirror.47256.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.39959.log.gz 2024-09-14T06:10:19.502 INFO:teuthology.orchestra.run.smithi191.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63538.log.gz 2024-09-14T06:10:19.502 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45048.log.gz 2024-09-14T06:10:19.502 INFO:teuthology.orchestra.run.smithi191.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.63869.log 2024-09-14T06:10:19.502 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67615.log 2024-09-14T06:10:19.503 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48514.log: 85.3% -- replaced with /var/log/ceph/cluster2-client.mirror.48514.log.gz 2024-09-14T06:10:19.503 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.63869.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41482.log 2024-09-14T06:10:19.503 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.67615.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.67615.log.gz 2024-09-14T06:10:19.503 INFO:teuthology.orchestra.run.smithi191.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63869.log.gz 2024-09-14T06:10:19.503 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47601.log 2024-09-14T06:10:19.503 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.41482.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60183.log 2024-09-14T06:10:19.503 INFO:teuthology.orchestra.run.smithi191.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.41482.log.gz 2024-09-14T06:10:19.514 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47601.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56987.log 2024-09-14T06:10:19.517 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.60183.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60183.log.gz 2024-09-14T06:10:19.524 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50831.log 2024-09-14T06:10:19.524 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.56987.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56987.log.gz 2024-09-14T06:10:19.525 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53865.log 2024-09-14T06:10:19.526 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.50831.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50831.log.gz 2024-09-14T06:10:19.526 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67098.log 2024-09-14T06:10:19.531 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.53865.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53865.log.gz 2024-09-14T06:10:19.531 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62622.log 2024-09-14T06:10:19.532 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.67098.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67098.log.gz 2024-09-14T06:10:19.536 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60042.log 2024-09-14T06:10:19.540 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.62622.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.62622.log.gz 2024-09-14T06:10:19.545 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55868.log 2024-09-14T06:10:19.547 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.60042.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60042.log.gz 2024-09-14T06:10:19.554 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48179.log 2024-09-14T06:10:19.555 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.55868.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.55868.log.gz 2024-09-14T06:10:19.555 INFO:teuthology.orchestra.run.smithi191.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46744.log.gz 2024-09-14T06:10:19.558 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59456.log 2024-09-14T06:10:19.559 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48179.log: 80.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48179.log.gz 2024-09-14T06:10:19.559 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50513.log 2024-09-14T06:10:19.562 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.59456.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.59456.log.gz 2024-09-14T06:10:19.562 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54485.log 2024-09-14T06:10:19.567 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.50513.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.50513.log.gz 2024-09-14T06:10:19.567 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37351.log 2024-09-14T06:10:19.571 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.54485.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54485.log.gz 2024-09-14T06:10:19.571 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41206.log 2024-09-14T06:10:19.572 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.37351.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56106.log 2024-09-14T06:10:19.577 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.41206.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.41206.log.gz 57.7% -- replaced with /var/log/ceph/cluster1-client.mirror.37351.log.gz 2024-09-14T06:10:19.577 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-14T06:10:19.577 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52242.log 2024-09-14T06:10:19.584 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.56106.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56106.log.gz 2024-09-14T06:10:19.585 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57705.log 2024-09-14T06:10:19.589 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.52242.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52242.log.gz 2024-09-14T06:10:19.589 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36365.log 2024-09-14T06:10:19.589 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.57705.log: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.57705.log.gz 2024-09-14T06:10:19.590 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67255.log 2024-09-14T06:10:19.590 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.admin.36365.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36365.log.gz 2024-09-14T06:10:19.599 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65023.log 2024-09-14T06:10:19.603 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.67255.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67255.log.gz 2024-09-14T06:10:19.608 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39829.log 2024-09-14T06:10:19.612 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65023.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65023.log.gz 2024-09-14T06:10:19.613 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40528.log 2024-09-14T06:10:19.616 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.39829.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47621.log 2024-09-14T06:10:19.617 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.40528.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.40528.log.gz 2024-09-14T06:10:19.618 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72161.log 2024-09-14T06:10:19.622 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47621.log: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47621.log.gz 2024-09-14T06:10:19.622 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61498.log 2024-09-14T06:10:19.622 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.72161.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72161.log.gz 2024-09-14T06:10:19.625 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70010.log 2024-09-14T06:10:19.626 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.61498.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61498.log.gz 2024-09-14T06:10:19.626 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49177.log 2024-09-14T06:10:19.635 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.70010.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54075.log 2024-09-14T06:10:19.640 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49177.log: 89.6% 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.70010.log.gz 2024-09-14T06:10:19.641 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.49177.log.gz 2024-09-14T06:10:19.641 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62575.log 2024-09-14T06:10:19.641 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.54075.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.54075.log.gz 2024-09-14T06:10:19.644 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64117.log 2024-09-14T06:10:19.655 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.62575.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66175.log 2024-09-14T06:10:19.655 INFO:teuthology.orchestra.run.smithi191.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47010.log.gz 2024-09-14T06:10:19.655 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64117.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68849.log 2024-09-14T06:10:19.656 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.66175.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66175.log.gz 2024-09-14T06:10:19.656 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63952.log 2024-09-14T06:10:19.657 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.68849.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.68849.log.gz 2024-09-14T06:10:19.657 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65831.log 2024-09-14T06:10:19.658 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.63952.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63952.log.gz 2024-09-14T06:10:19.658 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64262.log 2024-09-14T06:10:19.661 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65831.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65831.log.gz 2024-09-14T06:10:19.661 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65769.log 2024-09-14T06:10:19.662 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64262.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64262.log.gz 2024-09-14T06:10:19.663 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68290.log 2024-09-14T06:10:19.663 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65769.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.62575.log.gz 2024-09-14T06:10:19.666 INFO:teuthology.orchestra.run.smithi191.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65769.log.gz 2024-09-14T06:10:19.666 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39197.log 2024-09-14T06:10:19.669 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.68290.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68290.log.gz 2024-09-14T06:10:19.669 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54164.log 2024-09-14T06:10:19.669 INFO:teuthology.orchestra.run.smithi191.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.64117.log.gz 2024-09-14T06:10:19.669 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.39197.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44558.log 2024-09-14T06:10:19.669 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.54164.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60782.log 2024-09-14T06:10:19.670 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.44558.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44558.log.gz 2024-09-14T06:10:19.670 INFO:teuthology.orchestra.run.smithi191.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.39197.log.gz 2024-09-14T06:10:19.671 INFO:teuthology.orchestra.run.smithi191.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.54164.log.gz 2024-09-14T06:10:19.672 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70968.log 2024-09-14T06:10:19.674 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.60782.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.60782.log.gz 2024-09-14T06:10:19.675 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45916.log 2024-09-14T06:10:19.676 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.70968.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.70968.log.gz 2024-09-14T06:10:19.676 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50624.log 2024-09-14T06:10:19.682 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45916.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44263.log 2024-09-14T06:10:19.683 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.50624.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50624.log.gz 2024-09-14T06:10:19.683 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53190.log 2024-09-14T06:10:19.697 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.44263.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35609.log 2024-09-14T06:10:19.698 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.53190.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53190.log.gz 2024-09-14T06:10:19.711 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65955.log 2024-09-14T06:10:19.712 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.35609.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35609.log.gz 2024-09-14T06:10:19.712 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50738.log 2024-09-14T06:10:19.719 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.65955.log: 90.3%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67143.log 2024-09-14T06:10:19.719 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.65955.log.gz 2024-09-14T06:10:19.730 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.50738.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44263.log.gz 2024-09-14T06:10:19.732 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55909.log 2024-09-14T06:10:19.732 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.67143.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67143.log.gz 2024-09-14T06:10:19.734 INFO:teuthology.orchestra.run.smithi191.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.50738.log.gz 2024-09-14T06:10:19.734 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53415.log 2024-09-14T06:10:19.743 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.55909.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55909.log.gz 2024-09-14T06:10:19.743 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53502.log 2024-09-14T06:10:19.744 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.53415.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53415.log.gz 2024-09-14T06:10:19.744 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55796.log 2024-09-14T06:10:19.745 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.53502.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53502.log.gz 2024-09-14T06:10:19.745 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44818.log 2024-09-14T06:10:19.753 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.55796.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.55796.log.gz 2024-09-14T06:10:19.753 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32500.log 2024-09-14T06:10:19.754 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.44818.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44818.log.gz 2024-09-14T06:10:19.754 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60894.log 2024-09-14T06:10:19.757 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.admin.32500.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32500.log.gz 2024-09-14T06:10:19.758 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71832.log 2024-09-14T06:10:19.762 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.60894.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.60894.log.gz 2024-09-14T06:10:19.762 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68133.log 2024-09-14T06:10:19.771 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.71832.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68337.log 2024-09-14T06:10:19.771 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.68133.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68133.log.gz 2024-09-14T06:10:19.776 INFO:teuthology.orchestra.run.smithi191.stderr: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.71832.log.gz 2024-09-14T06:10:19.776 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49943.log 2024-09-14T06:10:19.777 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.68337.log: 2.6% -- replaced with /var/log/ceph/cluster1-client.mirror.68337.log.gz 2024-09-14T06:10:19.777 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41989.log 2024-09-14T06:10:19.781 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49943.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49943.log.gz 2024-09-14T06:10:19.785 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70500.log 2024-09-14T06:10:19.787 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.41989.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.41989.log.gz 2024-09-14T06:10:19.788 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.49700.log 2024-09-14T06:10:19.790 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.70500.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.70500.log.gz 2024-09-14T06:10:19.790 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39625.log 2024-09-14T06:10:19.795 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.2.49700.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.2.49700.log.gz 2024-09-14T06:10:19.795 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47047.log 2024-09-14T06:10:19.796 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.39625.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62788.log 2024-09-14T06:10:19.797 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47047.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47047.log.gz 2024-09-14T06:10:19.798 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64138.log 2024-09-14T06:10:19.798 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.62788.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62788.log.gz 2024-09-14T06:10:19.799 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63766.log 2024-09-14T06:10:19.803 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64138.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.39625.log.gz 2024-09-14T06:10:19.803 INFO:teuthology.orchestra.run.smithi191.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64138.log.gz 2024-09-14T06:10:19.804 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46619.log 2024-09-14T06:10:19.804 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.63766.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63766.log.gz 2024-09-14T06:10:19.811 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64386.log 2024-09-14T06:10:19.813 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46619.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46619.log.gz 2024-09-14T06:10:19.813 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46423.log 2024-09-14T06:10:19.818 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.64386.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64386.log.gz 2024-09-14T06:10:19.819 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44538.log 2024-09-14T06:10:19.848 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.46423.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.70440.log.gz 2024-09-14T06:10:19.849 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42150.log 2024-09-14T06:10:19.851 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.44538.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.44538.log.gz 2024-09-14T06:10:19.851 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45127.log 2024-09-14T06:10:19.857 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.42150.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.42150.log.gz 2024-09-14T06:10:19.857 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47441.log 2024-09-14T06:10:19.871 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.45127.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.42987.log 2024-09-14T06:10:19.878 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.47441.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56799.log 2024-09-14T06:10:19.882 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.0.42987.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.39829.log.gz 2024-09-14T06:10:19.885 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43661.log 2024-09-14T06:10:19.892 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.56799.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49923.log 2024-09-14T06:10:19.893 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.43661.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.56799.log.gz 2024-09-14T06:10:19.899 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37646.log 2024-09-14T06:10:19.906 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.49923.log: 89.6% 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.71555.log.gz 2024-09-14T06:10:19.910 INFO:teuthology.orchestra.run.smithi191.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43661.log.gz 2024-09-14T06:10:19.914 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.49923.log.gz 2024-09-14T06:10:19.924 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67233.log 2024-09-14T06:10:19.924 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.37646.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56519.log 2024-09-14T06:10:19.924 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.67233.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67233.log.gz 2024-09-14T06:10:19.925 INFO:teuthology.orchestra.run.smithi191.stderr: 80.6% -- replaced with /var/log/ceph/cluster2-client.mirror.37646.log.gz 2024-09-14T06:10:19.937 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40203.log 2024-09-14T06:10:19.937 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.56519.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.56519.log.gz 2024-09-14T06:10:19.951 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44758.log 2024-09-14T06:10:19.953 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.40203.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47680.log.gz 2024-09-14T06:10:19.953 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58612.log 2024-09-14T06:10:19.955 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.44758.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44758.log.gz 2024-09-14T06:10:19.955 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42987.log 2024-09-14T06:10:19.956 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.58612.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58612.log.gz 2024-09-14T06:10:19.956 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38334.log 2024-09-14T06:10:19.970 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.42987.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48238.log 2024-09-14T06:10:19.977 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.38334.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38866.log 2024-09-14T06:10:19.990 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster2-client.mirror.48238.log: 93.8%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.57705.log 2024-09-14T06:10:20.008 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/cluster1-client.mirror.38866.log: -- replaced with /var/log/ceph/cluster2-client.mirror.47601.log.gz 2024-09-14T06:10:20.027 INFO:teuthology.orchestra.run.smithi191.stderr: 93.7%/var/log/ceph/cluster2-client.mirror.0.57705.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.0.57705.log.gz 2024-09-14T06:10:20.027 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.47256.log.gz 2024-09-14T06:10:20.059 INFO:teuthology.orchestra.run.smithi191.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.42987.log.gz 2024-09-14T06:10:20.067 INFO:teuthology.orchestra.run.smithi191.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.0.42987.log.gz 2024-09-14T06:10:20.132 INFO:teuthology.orchestra.run.smithi191.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45916.log.gz 2024-09-14T06:10:20.305 INFO:teuthology.orchestra.run.smithi191.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.38866.log.gz 2024-09-14T06:10:20.312 INFO:teuthology.orchestra.run.smithi191.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46423.log.gz 2024-09-14T06:10:20.319 INFO:teuthology.orchestra.run.smithi191.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.40203.log.gz 2024-09-14T06:10:20.360 INFO:teuthology.orchestra.run.smithi191.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47441.log.gz 2024-09-14T06:10:20.360 INFO:teuthology.orchestra.run.smithi191.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45127.log.gz 2024-09-14T06:10:20.361 INFO:teuthology.orchestra.run.smithi191.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48238.log.gz 2024-09-14T06:10:20.617 INFO:teuthology.orchestra.run.smithi191.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-mon.a.log.gz 2024-09-14T06:10:23.179 INFO:teuthology.orchestra.run.smithi143.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-osd.1.log.gz 2024-09-14T06:10:24.031 INFO:teuthology.orchestra.run.smithi191.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.39873.log.gz 2024-09-14T06:10:24.497 INFO:teuthology.orchestra.run.smithi191.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.38334.log.gz 2024-09-14T06:10:24.977 INFO:teuthology.orchestra.run.smithi191.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.40379.log.gz 2024-09-14T06:10:25.256 INFO:teuthology.orchestra.run.smithi143.stderr: 94.5% -- replaced with /var/log/ceph/cluster1-osd.2.log.gz 2024-09-14T06:10:25.453 INFO:teuthology.orchestra.run.smithi143.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-osd.0.log.gz 2024-09-14T06:10:25.454 INFO:teuthology.orchestra.run.smithi143.stderr: 2024-09-14T06:10:25.455 INFO:teuthology.orchestra.run.smithi143.stderr:real 0m9.950s 2024-09-14T06:10:25.455 INFO:teuthology.orchestra.run.smithi143.stderr:user 0m27.777s 2024-09-14T06:10:25.455 INFO:teuthology.orchestra.run.smithi143.stderr:sys 0m1.071s 2024-09-14T06:11:41.322 INFO:teuthology.orchestra.run.smithi191.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-osd.1.log.gz 2024-09-14T06:11:44.913 INFO:teuthology.orchestra.run.smithi191.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-osd.2.log.gz 2024-09-14T06:12:07.444 INFO:teuthology.orchestra.run.smithi191.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-osd.0.log.gz 2024-09-14T06:12:07.446 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-14T06:12:07.446 INFO:teuthology.orchestra.run.smithi191.stderr:real 1m51.910s 2024-09-14T06:12:07.447 INFO:teuthology.orchestra.run.smithi191.stderr:user 3m50.804s 2024-09-14T06:12:07.447 INFO:teuthology.orchestra.run.smithi191.stderr:sys 0m9.943s 2024-09-14T06:12:07.447 INFO:tasks.ceph:Archiving logs... 2024-09-14T06:12:07.447 DEBUG:teuthology.misc:Transferring archived files from smithi143:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-10_20:16:17-rbd-main-distro-default-smithi/7900324/remote/smithi143/log 2024-09-14T06:12:07.449 DEBUG:teuthology.orchestra.run.smithi143:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-14T06:12:11.392 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-10_20:16:17-rbd-main-distro-default-smithi/7900324/remote/smithi191/log 2024-09-14T06:12:11.393 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-14T06:12:47.157 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-14T06:12:47.168 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-14T06:12:47.169 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-14T06:12:47.172 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-14T06:12:47.276 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-14T06:12:47.287 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-14T06:12:47.287 DEBUG:teuthology.orchestra.run.smithi143:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-14T06:12:47.290 DEBUG:teuthology.orchestra.run.smithi191:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-14T06:12:47.304 INFO:teuthology.orchestra.run.smithi143.stderr:bash: line 1: ntpq: command not found 2024-09-14T06:12:47.307 INFO:teuthology.orchestra.run.smithi191.stderr:bash: line 1: ntpq: command not found 2024-09-14T06:12:47.309 INFO:teuthology.orchestra.run.smithi143.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-14T06:12:47.309 INFO:teuthology.orchestra.run.smithi143.stdout:=============================================================================== 2024-09-14T06:12:47.309 INFO:teuthology.orchestra.run.smithi143.stdout:^* hv01.front.sepia.ceph.com 3 6 377 63 -356us[ -355us] +/- 47ms 2024-09-14T06:12:47.309 INFO:teuthology.orchestra.run.smithi143.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 64 +561us[ +562us] +/- 64ms 2024-09-14T06:12:47.309 INFO:teuthology.orchestra.run.smithi143.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 61 -28us[ -28us] +/- 81ms 2024-09-14T06:12:47.309 INFO:teuthology.orchestra.run.smithi143.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-14T06:12:47.337 INFO:teuthology.orchestra.run.smithi191.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-14T06:12:47.337 INFO:teuthology.orchestra.run.smithi191.stdout:=============================================================================== 2024-09-14T06:12:47.337 INFO:teuthology.orchestra.run.smithi191.stdout:^* hv01.front.sepia.ceph.com 3 6 377 64 -368us[ -366us] +/- 47ms 2024-09-14T06:12:47.338 INFO:teuthology.orchestra.run.smithi191.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 63 +538us[ +538us] +/- 64ms 2024-09-14T06:12:47.338 INFO:teuthology.orchestra.run.smithi191.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 61 -48us[ -48us] +/- 81ms 2024-09-14T06:12:47.338 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-14T06:12:47.338 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-14T06:12:47.368 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-14T06:12:47.369 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-14T06:12:47.554 DEBUG:teuthology.orchestra.run.smithi143:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-14T06:12:47.624 DEBUG:teuthology.orchestra.run.smithi191:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-14T06:12:47.711 DEBUG:teuthology.orchestra.run.smithi143:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-14T06:12:47.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T06:12:47.742 DEBUG:teuthology.orchestra.run.smithi191:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-14T06:12:47.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T06:12:47.773 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-14T06:12:47.786 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-14T06:12:47.979 INFO:teuthology.task.internal:Duration was 2814.085809 seconds 2024-09-14T06:12:47.980 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-14T06:12:48.077 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-14T06:12:48.078 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-14T06:12:48.081 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-14T06:12:48.124 INFO:teuthology.orchestra.run.smithi143.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-14T06:12:48.158 INFO:teuthology.orchestra.run.smithi191.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-14T06:12:48.660 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-14T06:12:48.661 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi143.front.sepia.ceph.com 2024-09-14T06:12:48.661 DEBUG:teuthology.orchestra.run.smithi143:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-14T06:12:48.685 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi191.front.sepia.ceph.com 2024-09-14T06:12:48.686 DEBUG:teuthology.orchestra.run.smithi191:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-14T06:12:48.709 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-14T06:12:48.709 DEBUG:teuthology.orchestra.run.smithi143:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-14T06:12:48.728 DEBUG:teuthology.orchestra.run.smithi191:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-14T06:12:48.778 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-14T06:12:48.779 DEBUG:teuthology.orchestra.run.smithi143:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-14T06:12:48.799 DEBUG:teuthology.orchestra.run.smithi191:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-14T06:12:48.931 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-14T06:12:48.952 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-14T06:12:48.953 DEBUG:teuthology.orchestra.run.smithi143:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-14T06:12:48.980 DEBUG:teuthology.orchestra.run.smithi191:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-14T06:12:49.016 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-14T06:12:49.099 DEBUG:teuthology.orchestra.run.smithi143:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-14T06:12:49.102 DEBUG:teuthology.orchestra.run.smithi191:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-14T06:12:49.123 INFO:teuthology.orchestra.run.smithi143.stdout:kernel.core_pattern = core 2024-09-14T06:12:49.137 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern = core 2024-09-14T06:12:49.159 DEBUG:teuthology.orchestra.run.smithi143:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-14T06:12:49.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T06:12:49.205 DEBUG:teuthology.orchestra.run.smithi191:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-14T06:12:49.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-14T06:12:49.220 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-14T06:12:49.235 INFO:teuthology.task.internal:Transferring archived files... 2024-09-14T06:12:49.236 DEBUG:teuthology.misc:Transferring archived files from smithi143:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-10_20:16:17-rbd-main-distro-default-smithi/7900324/remote/smithi143 2024-09-14T06:12:49.237 DEBUG:teuthology.orchestra.run.smithi143:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-14T06:12:49.281 DEBUG:teuthology.misc:Transferring archived files from smithi191:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-10_20:16:17-rbd-main-distro-default-smithi/7900324/remote/smithi191 2024-09-14T06:12:49.281 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-14T06:12:49.314 INFO:teuthology.task.internal:Removing archive directory... 2024-09-14T06:12:49.314 DEBUG:teuthology.orchestra.run.smithi143:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-14T06:12:49.317 DEBUG:teuthology.orchestra.run.smithi191:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-14T06:12:49.364 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-14T06:12:49.378 INFO:teuthology.task.internal:Not uploading archives. 2024-09-14T06:12:49.378 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-14T06:12:49.483 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-14T06:12:49.483 DEBUG:teuthology.orchestra.run.smithi143:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-14T06:12:49.486 DEBUG:teuthology.orchestra.run.smithi191:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-14T06:12:49.498 INFO:teuthology.orchestra.run.smithi143.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 14 06:12 /home/ubuntu/cephtest 2024-09-14T06:12:49.500 INFO:teuthology.orchestra.run.smithi191.stdout: 265836 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 14 06:12 /home/ubuntu/cephtest 2024-09-14T06:12:49.502 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-14T06:12:49.629 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-14T06:12:49.886 INFO:teuthology.run:Summary data: description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zstd policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} duration: 2814.085808992386 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-14T06:12:49.886 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-14T06:12:49.969 INFO:teuthology.run:pass